[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250902 01:56:44.951763 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:46173
I20250902 01:56:44.952983 29783 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250902 01:56:44.953817 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:56:44.968963 29789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:56:44.969096 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:56:44.969914 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:44.970281 29783 server_base.cc:1047] running on GCE node
I20250902 01:56:46.032727 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:56:46.032924 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:56:46.033102 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778206033072 us; error 0 us; skew 500 ppm
I20250902 01:56:46.033828 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:56:46.038515 29783 webserver.cc:480] Webserver started at http://127.29.21.254:36727/ using document root <none> and password file <none>
I20250902 01:56:46.039250 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:56:46.039428 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:56:46.039817 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:56:46.045259 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/master-0-root/instance:
uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.051887 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250902 01:56:46.056217 29798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.057307 29783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250902 01:56:46.057617 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/master-0-root
uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.057940 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:56:46.110347 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:56:46.111843 29783 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250902 01:56:46.112275 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:56:46.178018 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:46173
I20250902 01:56:46.178068 29849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.254:46173 every 8 connection(s)
I20250902 01:56:46.184288 29850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.203658 29850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: Bootstrap starting.
I20250902 01:56:46.208889 29850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:46.210561 29850 log.cc:826] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: Log is configured to *not* fsync() on all Append() calls
I20250902 01:56:46.214476 29850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: No bootstrap required, opened a new log
I20250902 01:56:46.232332 29850 raft_consensus.cc:357] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER }
I20250902 01:56:46.232824 29850 raft_consensus.cc:383] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:46.233057 29850 raft_consensus.cc:738] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0be53f1efc148e6a9b2a8b2d31bd2ac, State: Initialized, Role: FOLLOWER
I20250902 01:56:46.233732 29850 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER }
I20250902 01:56:46.234171 29850 raft_consensus.cc:397] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250902 01:56:46.234422 29850 raft_consensus.cc:491] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250902 01:56:46.234707 29850 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:46.238461 29850 raft_consensus.cc:513] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER }
I20250902 01:56:46.239081 29850 leader_election.cc:304] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0be53f1efc148e6a9b2a8b2d31bd2ac; no voters:
I20250902 01:56:46.240658 29850 leader_election.cc:290] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20250902 01:56:46.240981 29853 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:46.242899 29853 raft_consensus.cc:695] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 1 LEADER]: Becoming Leader. State: Replica: e0be53f1efc148e6a9b2a8b2d31bd2ac, State: Running, Role: LEADER
I20250902 01:56:46.243691 29853 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER }
I20250902 01:56:46.244714 29850 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [sys.catalog]: configured and running, proceeding with master startup.
I20250902 01:56:46.253079 29854 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER } }
I20250902 01:56:46.253578 29854 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [sys.catalog]: This master's current role is: LEADER
I20250902 01:56:46.252990 29855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0be53f1efc148e6a9b2a8b2d31bd2ac. Latest consensus state: current_term: 1 leader_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0be53f1efc148e6a9b2a8b2d31bd2ac" member_type: VOTER } }
I20250902 01:56:46.253911 29855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [sys.catalog]: This master's current role is: LEADER
I20250902 01:56:46.257076 29861 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250902 01:56:46.267689 29861 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250902 01:56:46.270117 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250902 01:56:46.282296 29861 catalog_manager.cc:1349] Generated new cluster ID: 483cbc4ae5b343bfa4aa0e3408208cb5
I20250902 01:56:46.282553 29861 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250902 01:56:46.305510 29861 catalog_manager.cc:1372] Generated new certificate authority record
I20250902 01:56:46.307329 29861 catalog_manager.cc:1506] Loading token signing keys...
I20250902 01:56:46.322872 29861 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: Generated new TSK 0
I20250902 01:56:46.323849 29861 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250902 01:56:46.336490 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:56:46.344861 29872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:56:46.346161 29873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.349592 29783 server_base.cc:1047] running on GCE node
W20250902 01:56:46.349987 29876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.350814 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:56:46.351022 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:56:46.351200 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778206351182 us; error 0 us; skew 500 ppm
I20250902 01:56:46.351742 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:56:46.355517 29783 webserver.cc:480] Webserver started at http://127.29.21.193:46281/ using document root <none> and password file <none>
I20250902 01:56:46.356003 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:56:46.356161 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:56:46.356396 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:56:46.357473 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-0-root/instance:
uuid: "33359a61000f449ab9b7172f83c3c946"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.362574 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20250902 01:56:46.366803 29880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.367722 29783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250902 01:56:46.368042 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
uuid: "33359a61000f449ab9b7172f83c3c946"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.368410 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:56:46.386945 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:56:46.388574 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:56:46.391047 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:56:46.395571 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:56:46.395836 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250902 01:56:46.396118 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:56:46.396345 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.483887 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:42657
I20250902 01:56:46.483932 29942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.193:42657 every 8 connection(s)
I20250902 01:56:46.491158 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:56:46.504210 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:56:46.506244 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.512094 29783 server_base.cc:1047] running on GCE node
W20250902 01:56:46.513548 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.514621 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:56:46.514860 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:56:46.515053 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778206515038 us; error 0 us; skew 500 ppm
I20250902 01:56:46.515776 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:56:46.519883 29783 webserver.cc:480] Webserver started at http://127.29.21.194:40279/ using document root <none> and password file <none>
I20250902 01:56:46.520085 29943 heartbeater.cc:344] Connected to a master server at 127.29.21.254:46173
I20250902 01:56:46.520505 29943 heartbeater.cc:461] Registering TS with master...
I20250902 01:56:46.520517 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:56:46.520821 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:56:46.521126 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:56:46.521456 29943 heartbeater.cc:507] Master 127.29.21.254:46173 requested a full tablet report, sending...
I20250902 01:56:46.522554 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-1-root/instance:
uuid: "968ef8b8113743b7a82cb596076f6c11"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.523941 29815 ts_manager.cc:194] Registered new tserver with Master: 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657)
I20250902 01:56:46.525893 29815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35458
I20250902 01:56:46.529145 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250902 01:56:46.533658 29955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.534341 29783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250902 01:56:46.534621 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
uuid: "968ef8b8113743b7a82cb596076f6c11"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.534932 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:56:46.568665 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:56:46.570153 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:56:46.571508 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:56:46.574684 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:56:46.574895 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.575132 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:56:46.575305 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.654469 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:33815
I20250902 01:56:46.654529 30017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.194:33815 every 8 connection(s)
I20250902 01:56:46.661352 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:56:46.669157 30021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:56:46.671902 30022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.674178 30018 heartbeater.cc:344] Connected to a master server at 127.29.21.254:46173
I20250902 01:56:46.674615 30018 heartbeater.cc:461] Registering TS with master...
I20250902 01:56:46.675725 30018 heartbeater.cc:507] Master 127.29.21.254:46173 requested a full tablet report, sending...
I20250902 01:56:46.678172 29815 ts_manager.cc:194] Registered new tserver with Master: 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:46.678740 29783 server_base.cc:1047] running on GCE node
W20250902 01:56:46.679915 30024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:56:46.679937 29815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35472
I20250902 01:56:46.681000 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:56:46.681243 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:56:46.681427 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778206681411 us; error 0 us; skew 500 ppm
I20250902 01:56:46.682078 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:56:46.684515 29783 webserver.cc:480] Webserver started at http://127.29.21.195:43249/ using document root <none> and password file <none>
I20250902 01:56:46.685091 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:56:46.685329 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:56:46.685598 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:56:46.686825 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-2-root/instance:
uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.692056 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250902 01:56:46.695955 30029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.696710 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250902 01:56:46.696980 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
format_stamp: "Formatted at 2025-09-02 01:56:46 on dist-test-slave-g0c9"
I20250902 01:56:46.697213 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756778204804806-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:56:46.716702 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:56:46.717859 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:56:46.719120 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:56:46.721813 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:56:46.721995 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.722210 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:56:46.722357 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:56:46.805589 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:36599
I20250902 01:56:46.805732 30091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.195:36599 every 8 connection(s)
I20250902 01:56:46.820837 30092 heartbeater.cc:344] Connected to a master server at 127.29.21.254:46173
I20250902 01:56:46.821148 30092 heartbeater.cc:461] Registering TS with master...
I20250902 01:56:46.821707 30092 heartbeater.cc:507] Master 127.29.21.254:46173 requested a full tablet report, sending...
I20250902 01:56:46.823608 29815 ts_manager.cc:194] Registered new tserver with Master: 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:46.824631 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013184168s
I20250902 01:56:46.824803 29815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35478
I20250902 01:56:46.827335 29783 test_util.cc:276] Using random seed: -856190164
I20250902 01:56:46.867106 29815 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35488:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250902 01:56:46.870419 29815 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250902 01:56:46.964761 29908 tablet_service.cc:1468] Processing CreateTablet for tablet 1512fda84c5a470fbcf5b6080b4ab3ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250902 01:56:46.966358 29908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1512fda84c5a470fbcf5b6080b4ab3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.969259 29907 tablet_service.cc:1468] Processing CreateTablet for tablet 3344310a48194896aacca0abcdee4e52 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250902 01:56:46.970562 29907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3344310a48194896aacca0abcdee4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.972481 30057 tablet_service.cc:1468] Processing CreateTablet for tablet 1512fda84c5a470fbcf5b6080b4ab3ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250902 01:56:46.973577 30057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1512fda84c5a470fbcf5b6080b4ab3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.975018 30056 tablet_service.cc:1468] Processing CreateTablet for tablet 3344310a48194896aacca0abcdee4e52 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250902 01:56:46.976279 30056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3344310a48194896aacca0abcdee4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.976790 30054 tablet_service.cc:1468] Processing CreateTablet for tablet bd26046f3ae04db7ae1542964786ceb5 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250902 01:56:46.978150 30054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd26046f3ae04db7ae1542964786ceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.980733 30049 tablet_service.cc:1468] Processing CreateTablet for tablet 74024619edd643b492526702db04fa94 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250902 01:56:46.981410 30052 tablet_service.cc:1468] Processing CreateTablet for tablet 7375387eca6f4ec1886e409e60131330 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250902 01:56:46.982044 30049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74024619edd643b492526702db04fa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.983385 30050 tablet_service.cc:1468] Processing CreateTablet for tablet 53a0165266f441bdaba7be5d35c7e58a (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250902 01:56:46.985409 30050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a0165266f441bdaba7be5d35c7e58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.990725 30052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7375387eca6f4ec1886e409e60131330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:46.991380 29906 tablet_service.cc:1468] Processing CreateTablet for tablet bd26046f3ae04db7ae1542964786ceb5 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250902 01:56:46.987421 30053 tablet_service.cc:1468] Processing CreateTablet for tablet 184c16e4c06d492eba1c49f5ccad2d88 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250902 01:56:47.010828 29906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd26046f3ae04db7ae1542964786ceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.010607 30051 tablet_service.cc:1468] Processing CreateTablet for tablet f17b201ff2784ad08192abc643e592ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250902 01:56:47.001350 29983 tablet_service.cc:1468] Processing CreateTablet for tablet 1512fda84c5a470fbcf5b6080b4ab3ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250902 01:56:47.010135 30055 tablet_service.cc:1468] Processing CreateTablet for tablet 746252cfaffb40d0b9f70ad6a0b4a56c (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250902 01:56:47.021317 30053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184c16e4c06d492eba1c49f5ccad2d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.024772 29981 tablet_service.cc:1468] Processing CreateTablet for tablet bd26046f3ae04db7ae1542964786ceb5 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250902 01:56:47.026719 30051 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b201ff2784ad08192abc643e592ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.035898 30055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746252cfaffb40d0b9f70ad6a0b4a56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.055639 29983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1512fda84c5a470fbcf5b6080b4ab3ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.064846 29902 tablet_service.cc:1468] Processing CreateTablet for tablet f17b201ff2784ad08192abc643e592ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250902 01:56:47.026713 29905 tablet_service.cc:1468] Processing CreateTablet for tablet 746252cfaffb40d0b9f70ad6a0b4a56c (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250902 01:56:47.057384 29980 tablet_service.cc:1468] Processing CreateTablet for tablet 746252cfaffb40d0b9f70ad6a0b4a56c (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250902 01:56:47.050333 29904 tablet_service.cc:1468] Processing CreateTablet for tablet 184c16e4c06d492eba1c49f5ccad2d88 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250902 01:56:47.068158 29902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b201ff2784ad08192abc643e592ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.041503 29982 tablet_service.cc:1468] Processing CreateTablet for tablet 3344310a48194896aacca0abcdee4e52 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250902 01:56:47.075906 29904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184c16e4c06d492eba1c49f5ccad2d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.073506 29981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd26046f3ae04db7ae1542964786ceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.070392 29903 tablet_service.cc:1468] Processing CreateTablet for tablet 7375387eca6f4ec1886e409e60131330 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250902 01:56:47.078948 29905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746252cfaffb40d0b9f70ad6a0b4a56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.092758 29908 tablet_service.cc:1468] Processing CreateTablet for tablet 53a0165266f441bdaba7be5d35c7e58a (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250902 01:56:47.093140 29903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7375387eca6f4ec1886e409e60131330. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.120107 29977 tablet_service.cc:1468] Processing CreateTablet for tablet f17b201ff2784ad08192abc643e592ff (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250902 01:56:47.093876 29980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746252cfaffb40d0b9f70ad6a0b4a56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.093263 30110 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.122260 29979 tablet_service.cc:1468] Processing CreateTablet for tablet 184c16e4c06d492eba1c49f5ccad2d88 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250902 01:56:47.124310 29908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a0165266f441bdaba7be5d35c7e58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.124082 29901 tablet_service.cc:1468] Processing CreateTablet for tablet 74024619edd643b492526702db04fa94 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250902 01:56:47.130239 29982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3344310a48194896aacca0abcdee4e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.128886 29976 tablet_service.cc:1468] Processing CreateTablet for tablet 53a0165266f441bdaba7be5d35c7e58a (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250902 01:56:47.130172 29975 tablet_service.cc:1468] Processing CreateTablet for tablet 74024619edd643b492526702db04fa94 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250902 01:56:47.124749 29978 tablet_service.cc:1468] Processing CreateTablet for tablet 7375387eca6f4ec1886e409e60131330 (DEFAULT_TABLE table=test-workload [id=24c9ea0c60494645bb1df98766374e70]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250902 01:56:47.141069 29901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74024619edd643b492526702db04fa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.167389 29977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b201ff2784ad08192abc643e592ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.170514 29979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184c16e4c06d492eba1c49f5ccad2d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.172309 29976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a0165266f441bdaba7be5d35c7e58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.182355 29975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74024619edd643b492526702db04fa94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:56:47.196058 29978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7375387eca6f4ec1886e409e60131330. 1 dirs total, 0 dirs full, 0 dirs failed
W20250902 01:56:47.296716 29863 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:56:47.299876 30111 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:47.330430 30111 tablet_bootstrap.cc:654] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.368427 30111 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:47.368850 30111 ts_tablet_manager.cc:1397] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.007s
I20250902 01:56:47.371994 30110 tablet_bootstrap.cc:654] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.383807 30112 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.399669 30110 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.400473 30110 ts_tablet_manager.cc:1397] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.350s user 0.003s sys 0.017s
I20250902 01:56:47.403191 30111 raft_consensus.cc:357] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.404104 30111 raft_consensus.cc:383] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.404409 30111 raft_consensus.cc:738] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.404379 30110 raft_consensus.cc:357] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.405174 30110 raft_consensus.cc:383] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.405462 30110 raft_consensus.cc:738] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.405308 30111 consensus_queue.cc:260] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.406073 30110 consensus_queue.cc:260] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.410506 29943 heartbeater.cc:499] Master 127.29.21.254:46173 was elected leader, sending a full tablet report...
I20250902 01:56:47.416939 30111 ts_tablet_manager.cc:1428] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.048s user 0.005s sys 0.008s
I20250902 01:56:47.417129 30112 tablet_bootstrap.cc:654] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.418602 30111 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:47.421231 30110 ts_tablet_manager.cc:1428] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250902 01:56:47.422049 30110 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.423799 30092 heartbeater.cc:499] Master 127.29.21.254:46173 was elected leader, sending a full tablet report...
I20250902 01:56:47.425189 30111 tablet_bootstrap.cc:654] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.432394 30112 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.432772 30112 ts_tablet_manager.cc:1397] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.010s
I20250902 01:56:47.435401 30112 raft_consensus.cc:357] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.437249 30110 tablet_bootstrap.cc:654] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.436142 30112 raft_consensus.cc:383] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.439924 30112 raft_consensus.cc:738] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.440690 30112 consensus_queue.cc:260] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.452656 30110 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.453325 30018 heartbeater.cc:499] Master 127.29.21.254:46173 was elected leader, sending a full tablet report...
I20250902 01:56:47.454833 30112 ts_tablet_manager.cc:1428] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.022s user 0.005s sys 0.005s
I20250902 01:56:47.456596 30112 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.453467 30110 ts_tablet_manager.cc:1397] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.005s
I20250902 01:56:47.461624 30110 raft_consensus.cc:357] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.462230 30110 raft_consensus.cc:383] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.462481 30110 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.463037 30110 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.469094 30112 tablet_bootstrap.cc:654] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.470847 30110 ts_tablet_manager.cc:1428] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20250902 01:56:47.471647 30110 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.476660 30110 tablet_bootstrap.cc:654] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.487737 30112 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.488178 30112 ts_tablet_manager.cc:1397] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.005s
I20250902 01:56:47.490767 30112 raft_consensus.cc:357] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.491334 30112 raft_consensus.cc:383] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250902 01:56:47.492532 29944 tablet.cc:2378] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250902 01:56:47.492709 30112 raft_consensus.cc:738] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.493394 30112 consensus_queue.cc:260] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.495801 30110 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.496172 30110 ts_tablet_manager.cc:1397] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20250902 01:56:47.498624 30110 raft_consensus.cc:357] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.499182 30110 raft_consensus.cc:383] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.499426 30110 raft_consensus.cc:738] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.499997 30110 consensus_queue.cc:260] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.502115 30112 ts_tablet_manager.cc:1428] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250902 01:56:47.502511 30111 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:47.502846 30112 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.502883 30111 ts_tablet_manager.cc:1397] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.084s user 0.034s sys 0.046s
I20250902 01:56:47.505415 30111 raft_consensus.cc:357] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.506098 30111 raft_consensus.cc:383] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.506357 30111 raft_consensus.cc:738] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.507040 30111 consensus_queue.cc:260] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.509822 30111 ts_tablet_manager.cc:1428] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250902 01:56:47.510171 30110 ts_tablet_manager.cc:1428] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250902 01:56:47.510605 30111 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:47.510979 30110 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.516808 30110 tablet_bootstrap.cc:654] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.517204 30111 tablet_bootstrap.cc:654] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.520505 30112 tablet_bootstrap.cc:654] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.531996 30110 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.532382 30110 ts_tablet_manager.cc:1397] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20250902 01:56:47.534570 30110 raft_consensus.cc:357] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.535149 30110 raft_consensus.cc:383] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.535382 30110 raft_consensus.cc:738] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.535928 30110 consensus_queue.cc:260] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.538417 30110 ts_tablet_manager.cc:1428] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:56:47.539211 30110 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.544449 30110 tablet_bootstrap.cc:654] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.552723 30112 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.553123 30112 ts_tablet_manager.cc:1397] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
I20250902 01:56:47.553303 30111 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:47.553619 30111 ts_tablet_manager.cc:1397] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.020s
I20250902 01:56:47.555286 30111 raft_consensus.cc:357] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.555963 30111 raft_consensus.cc:383] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.555766 30112 raft_consensus.cc:357] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.556398 30112 raft_consensus.cc:383] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.556308 30111 raft_consensus.cc:738] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.556731 30112 raft_consensus.cc:738] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.557348 30111 consensus_queue.cc:260] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.557529 30112 consensus_queue.cc:260] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.559795 30111 ts_tablet_manager.cc:1428] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250902 01:56:47.559859 30112 ts_tablet_manager.cc:1428] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20250902 01:56:47.560673 30111 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:47.560896 30112 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.566526 30112 tablet_bootstrap.cc:654] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.568298 30111 tablet_bootstrap.cc:654] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.579434 30110 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.580085 30110 ts_tablet_manager.cc:1397] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.007s
I20250902 01:56:47.582376 30112 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.582752 30112 ts_tablet_manager.cc:1397] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.012s
I20250902 01:56:47.583406 30110 raft_consensus.cc:357] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.584153 30110 raft_consensus.cc:383] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.584396 30110 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.584952 30110 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.585206 30112 raft_consensus.cc:357] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.586123 30111 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:47.586154 30112 raft_consensus.cc:383] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.586555 30112 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.586694 30111 ts_tablet_manager.cc:1397] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.010s
I20250902 01:56:47.587370 30116 raft_consensus.cc:491] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.587932 30116 raft_consensus.cc:513] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.594928 30110 ts_tablet_manager.cc:1428] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20250902 01:56:47.587419 30112 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.596043 30110 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.597074 30116 leader_election.cc:290] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:47.603080 30120 raft_consensus.cc:491] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.603520 30120 raft_consensus.cc:513] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.603910 30112 ts_tablet_manager.cc:1428] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.021s user 0.005s sys 0.008s
I20250902 01:56:47.604871 30112 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.605839 30120 leader_election.cc:290] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.599361 30111 raft_consensus.cc:357] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.612877 30111 raft_consensus.cc:383] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.613162 30111 raft_consensus.cc:738] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.618747 30111 consensus_queue.cc:260] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.629304 30110 tablet_bootstrap.cc:654] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.634418 30111 ts_tablet_manager.cc:1428] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.047s user 0.003s sys 0.003s
I20250902 01:56:47.637480 30115 raft_consensus.cc:491] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.637945 30115 raft_consensus.cc:513] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.639212 30111 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:47.644833 30115 leader_election.cc:290] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:47.654105 30112 tablet_bootstrap.cc:654] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.657012 30116 raft_consensus.cc:491] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.658540 30116 raft_consensus.cc:513] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.662339 30116 leader_election.cc:290] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.670830 30110 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.671254 30110 ts_tablet_manager.cc:1397] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.075s user 0.015s sys 0.001s
I20250902 01:56:47.690654 30112 tablet_bootstrap.cc:492] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.689857 30110 raft_consensus.cc:357] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.692076 30112 ts_tablet_manager.cc:1397] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.006s
I20250902 01:56:47.695576 30120 raft_consensus.cc:491] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.693199 30110 raft_consensus.cc:383] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.697419 30120 raft_consensus.cc:513] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.698350 30110 raft_consensus.cc:738] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.708664 30112 raft_consensus.cc:357] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.709774 30112 raft_consensus.cc:383] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.710045 30112 raft_consensus.cc:738] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.711421 30112 consensus_queue.cc:260] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.721383 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.715502 30120 leader_election.cc:290] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.725239 30111 tablet_bootstrap.cc:654] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.726037 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
W20250902 01:56:47.727871 29882 leader_election.cc:343] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.728039 30066 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:47.732435 29882 leader_election.cc:304] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:47.733163 30116 raft_consensus.cc:2802] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.722491 30110 consensus_queue.cc:260] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.733731 30116 raft_consensus.cc:491] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.736344 30116 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.738719 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.739099 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.739634 30067 raft_consensus.cc:2466] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 0.
W20250902 01:56:47.740957 29957 leader_election.cc:343] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.742698 29993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.744010 29993 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:47.746181 29957 leader_election.cc:304] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:47.746912 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.748986 30120 raft_consensus.cc:2802] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.748978 29992 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:47.749356 30120 raft_consensus.cc:491] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.749678 30120 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.747023 29883 leader_election.cc:304] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:47.765128 30120 raft_consensus.cc:513] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.775211 30116 raft_consensus.cc:513] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.777256 30116 leader_election.cc:290] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.778971 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:47.779453 30067 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.781692 30112 ts_tablet_manager.cc:1428] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.087s user 0.005s sys 0.006s
I20250902 01:56:47.782603 30112 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.783756 30116 raft_consensus.cc:2802] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.789111 30120 leader_election.cc:290] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.790210 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:47.790833 30066 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.791805 30143 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.785864 30116 raft_consensus.cc:491] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.792845 30116 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.792732 30067 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 1.
I20250902 01:56:47.793927 30143 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.803278 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:47.806087 29882 leader_election.cc:304] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:47.811340 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:47.811997 30143 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.811767 30144 raft_consensus.cc:2802] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:47.812881 30144 raft_consensus.cc:695] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
W20250902 01:56:47.811301 29883 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:33815, user_credentials={real_user=slave}} blocked reactor thread for 64659us
I20250902 01:56:47.816275 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:47.816758 29992 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:56:47.819598 29958 leader_election.cc:343] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.820045 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:47.817804 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.819211 30110 ts_tablet_manager.cc:1428] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.147s user 0.010s sys 0.010s
I20250902 01:56:47.820372 29958 leader_election.cc:304] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946
W20250902 01:56:47.825700 29958 leader_election.cc:343] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:56:47.826447 29958 leader_election.cc:343] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.827893 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:47.874353 30111 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:47.834235 30112 tablet_bootstrap.cc:654] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.872665 29993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.837050 30143 raft_consensus.cc:2747] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:47.831700 30110 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.851992 29917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:47.867157 30115 raft_consensus.cc:491] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.830444 29992 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 1.
I20250902 01:56:47.876578 30111 ts_tablet_manager.cc:1397] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.238s user 0.021s sys 0.037s
I20250902 01:56:47.881392 30115 raft_consensus.cc:513] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.883041 30115 leader_election.cc:290] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
W20250902 01:56:47.829394 29957 leader_election.cc:343] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.883735 30116 raft_consensus.cc:513] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
W20250902 01:56:47.885898 29957 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:36599, user_credentials={real_user=slave}} blocked reactor thread for 56520.9us
I20250902 01:56:47.886832 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
W20250902 01:56:47.888104 29882 leader_election.cc:343] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.888964 30116 leader_election.cc:290] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Requested vote from peers 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:47.889487 30116 raft_consensus.cc:491] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.889943 30116 raft_consensus.cc:513] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.891603 30110 tablet_bootstrap.cc:654] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.892683 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:47.893411 30116 leader_election.cc:290] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.893899 30116 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.894461 30116 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.883713 30111 raft_consensus.cc:357] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.895114 29993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.895318 30111 raft_consensus.cc:383] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.895759 30111 raft_consensus.cc:738] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.863169 30066 raft_consensus.cc:2466] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 1.
I20250902 01:56:47.892715 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.896418 30067 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:47.896792 30112 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:47.896672 30111 consensus_queue.cc:260] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.897524 29882 leader_election.cc:304] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:47.893505 29992 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.898994 29957 leader_election.cc:304] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters: 33359a61000f449ab9b7172f83c3c946
I20250902 01:56:47.847267 30144 consensus_queue.cc:237] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.901726 30110 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:47.902143 30110 ts_tablet_manager.cc:1397] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.012s
I20250902 01:56:47.902302 30111 ts_tablet_manager.cc:1428] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20250902 01:56:47.904042 30147 raft_consensus.cc:2802] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.904244 30143 raft_consensus.cc:2802] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:47.897308 30112 ts_tablet_manager.cc:1397] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.115s user 0.011s sys 0.007s
I20250902 01:56:47.905483 30143 raft_consensus.cc:695] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:47.906327 30143 consensus_queue.cc:237] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.904775 30110 raft_consensus.cc:357] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.861794 30145 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:56:47.879737 30032 leader_election.cc:343] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.909281 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.904497 30111 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
W20250902 01:56:47.910331 29882 leader_election.cc:343] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.912940 30115 raft_consensus.cc:491] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:47.913481 30115 raft_consensus.cc:513] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.907683 30112 raft_consensus.cc:357] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.909281 30145 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.913930 29993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
W20250902 01:56:47.913985 30032 leader_election.cc:343] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.914598 29918 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 0.
I20250902 01:56:47.904477 30147 raft_consensus.cc:491] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.915257 30147 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.915467 30115 leader_election.cc:290] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:47.914763 30032 leader_election.cc:304] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4; no voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:47.914235 30112 raft_consensus.cc:383] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.916280 30112 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.917068 30112 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.917865 29958 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:47.916342 30116 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.913733 29917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
W20250902 01:56:47.907977 29883 leader_election.cc:343] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.909091 30110 raft_consensus.cc:383] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:47.919170 29992 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 1.
I20250902 01:56:47.922930 30120 raft_consensus.cc:2802] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.923224 30120 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.920238 30115 raft_consensus.cc:2747] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:47.924226 30112 ts_tablet_manager.cc:1428] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20250902 01:56:47.924769 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:47.920396 30110 raft_consensus.cc:738] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:47.930150 29918 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 0.
I20250902 01:56:47.930290 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:47.930425 30110 consensus_queue.cc:260] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.920360 30145 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
W20250902 01:56:47.931994 29882 leader_election.cc:343] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.923491 30120 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.935643 30112 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:47.935694 30147 raft_consensus.cc:513] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.938237 30110 ts_tablet_manager.cc:1428] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.036s user 0.003s sys 0.005s
W20250902 01:56:47.939184 30032 leader_election.cc:343] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.948297 30032 leader_election.cc:304] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:47.949005 30115 raft_consensus.cc:2802] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:47.949344 30115 raft_consensus.cc:491] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:47.949647 30115 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.949725 30110 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:47.951656 29917 raft_consensus.cc:2391] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in current term 1: Already voted for candidate 33359a61000f449ab9b7172f83c3c946 in this term.
I20250902 01:56:47.956998 30115 raft_consensus.cc:513] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.953181 30032 leader_election.cc:304] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4; no voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:47.963413 30147 leader_election.cc:290] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.959077 29815 catalog_manager.cc:5582] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:47.964226 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:47.965461 30066 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.961805 29814 catalog_manager.cc:5582] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:47.969179 30120 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.970196 30111 tablet_bootstrap.cc:654] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.973433 30112 tablet_bootstrap.cc:654] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:47.975616 30115 leader_election.cc:290] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:47.964226 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.962313 29883 leader_election.cc:304] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:47.979415 29993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.980305 29992 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:47.982728 29993 raft_consensus.cc:2391] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33359a61000f449ab9b7172f83c3c946 in current term 1: Already voted for candidate 968ef8b8113743b7a82cb596076f6c11 in this term.
I20250902 01:56:47.983227 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:47.982082 30147 raft_consensus.cc:2802] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:47.984711 30147 raft_consensus.cc:695] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:47.988830 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:47.989925 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:47.986871 30147 consensus_queue.cc:237] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:47.985950 30066 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 1.
I20250902 01:56:47.990327 29917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946"
W20250902 01:56:47.991715 29957 leader_election.cc:343] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.992013 30115 raft_consensus.cc:2747] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:47.992069 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:47.989149 29883 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:47.987893 30120 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:47.993031 29917 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.993790 29882 leader_election.cc:304] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:47.999466 30116 raft_consensus.cc:2802] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:47.999854 30116 raft_consensus.cc:695] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:48.000442 30116 consensus_queue.cc:237] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:47.993975 29918 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:47.996892 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:47.997725 29883 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11
W20250902 01:56:48.004062 29883 leader_election.cc:343] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:56:47.998495 30032 leader_election.cc:343] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:56:48.005331 30032 leader_election.cc:343] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:47.999022 30110 tablet_bootstrap.cc:654] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.006600 30144 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:48.007006 30144 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:48.007330 30144 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.017010 30112 tablet_bootstrap.cc:492] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:48.017432 30112 ts_tablet_manager.cc:1397] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.092s user 0.015s sys 0.006s
I20250902 01:56:48.018795 30111 tablet_bootstrap.cc:492] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:48.019227 30111 ts_tablet_manager.cc:1397] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.116s user 0.016s sys 0.016s
I20250902 01:56:48.031548 29918 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 1.
I20250902 01:56:48.033187 30144 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.033828 30145 raft_consensus.cc:2747] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:48.035399 30144 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.036080 29958 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:48.033169 29917 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 1.
I20250902 01:56:48.039219 30120 raft_consensus.cc:2802] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:48.039215 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:48.039736 30120 raft_consensus.cc:695] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:48.040020 29992 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.040429 30120 consensus_queue.cc:237] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.042999 30032 leader_election.cc:304] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters: 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.043905 30115 raft_consensus.cc:2802] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:48.044294 30115 raft_consensus.cc:695] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:56:48.045396 30115 consensus_queue.cc:237] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.039215 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:48.041733 30111 raft_consensus.cc:357] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:48.047394 30111 raft_consensus.cc:383] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.047772 30111 raft_consensus.cc:738] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
W20250902 01:56:48.046823 29882 leader_election.cc:343] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:56:48.048571 30111 consensus_queue.cc:260] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:48.045089 30112 raft_consensus.cc:357] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.056308 30112 raft_consensus.cc:383] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.056843 30112 raft_consensus.cc:738] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.057416 29992 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 1.
I20250902 01:56:48.057329 30112 consensus_queue.cc:260] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.058465 29883 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:48.059736 30147 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:48.062206 30111 ts_tablet_manager.cc:1428] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.043s user 0.001s sys 0.005s
I20250902 01:56:48.062980 30110 tablet_bootstrap.cc:492] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:48.063134 30111 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:48.063351 30110 ts_tablet_manager.cc:1397] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.115s user 0.018s sys 0.012s
I20250902 01:56:48.067101 30112 ts_tablet_manager.cc:1428] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.049s user 0.002s sys 0.006s
I20250902 01:56:48.068460 30112 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:48.071575 30147 raft_consensus.cc:695] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:48.075266 30147 consensus_queue.cc:237] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.074133 30110 raft_consensus.cc:357] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.077865 30110 raft_consensus.cc:383] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.078203 30110 raft_consensus.cc:738] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.074203 30111 tablet_bootstrap.cc:654] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.079010 30110 consensus_queue.cc:260] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.081671 30110 ts_tablet_manager.cc:1428] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250902 01:56:48.082417 30112 tablet_bootstrap.cc:654] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.082589 30110 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: Bootstrap starting.
I20250902 01:56:48.089437 30112 tablet_bootstrap.cc:492] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:48.090749 30112 ts_tablet_manager.cc:1397] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.008s
I20250902 01:56:48.092808 30112 raft_consensus.cc:357] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.093333 30112 raft_consensus.cc:383] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.093551 30112 raft_consensus.cc:738] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.093524 30120 raft_consensus.cc:491] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.094182 30120 raft_consensus.cc:513] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.094276 30112 consensus_queue.cc:260] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.095665 30110 tablet_bootstrap.cc:654] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.095906 30120 leader_election.cc:290] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.097265 30111 tablet_bootstrap.cc:492] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:48.097678 30111 ts_tablet_manager.cc:1397] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.006s
I20250902 01:56:48.098060 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.098865 29918 raft_consensus.cc:2391] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 968ef8b8113743b7a82cb596076f6c11 in current term 1: Already voted for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in this term.
I20250902 01:56:48.099797 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:48.100565 30111 raft_consensus.cc:357] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.101140 30111 raft_consensus.cc:383] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.101379 30111 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.102072 30111 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.102341 29958 leader_election.cc:304] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 968ef8b8113743b7a82cb596076f6c11; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946
I20250902 01:56:48.104260 30111 ts_tablet_manager.cc:1428] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250902 01:56:48.105127 30111 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:48.110788 30111 tablet_bootstrap.cc:654] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.114166 30120 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.118680 29813 catalog_manager.cc:5582] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:56:48.119832 30120 raft_consensus.cc:2747] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:48.121276 30112 ts_tablet_manager.cc:1428] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250902 01:56:48.121345 29814 catalog_manager.cc:5582] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.122042 30112 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: Bootstrap starting.
I20250902 01:56:48.122648 29814 catalog_manager.cc:5582] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.124610 29814 catalog_manager.cc:5582] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.126981 30112 tablet_bootstrap.cc:654] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.130301 30110 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: No bootstrap required, opened a new log
I20250902 01:56:48.137133 30110 ts_tablet_manager.cc:1397] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.003s
I20250902 01:56:48.132377 30111 tablet_bootstrap.cc:492] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:48.138760 30112 tablet_bootstrap.cc:492] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: No bootstrap required, opened a new log
I20250902 01:56:48.139101 30112 ts_tablet_manager.cc:1397] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250902 01:56:48.141701 29815 catalog_manager.cc:5582] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.148986 30111 ts_tablet_manager.cc:1397] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.003s
I20250902 01:56:48.150529 30110 raft_consensus.cc:357] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.151130 30110 raft_consensus.cc:383] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.151397 30110 raft_consensus.cc:738] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.151355 30112 raft_consensus.cc:357] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.151966 30112 raft_consensus.cc:383] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.152190 30110 consensus_queue.cc:260] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.152197 30112 raft_consensus.cc:738] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.153337 30111 raft_consensus.cc:357] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.153856 30111 raft_consensus.cc:383] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.154078 30111 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.154589 30111 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.156333 30111 ts_tablet_manager.cc:1428] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20250902 01:56:48.153504 30112 consensus_queue.cc:260] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.157073 30111 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Bootstrap starting.
I20250902 01:56:48.157747 30110 ts_tablet_manager.cc:1428] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: Time spent starting tablet: real 0.020s user 0.003s sys 0.011s
I20250902 01:56:48.159122 30112 ts_tablet_manager.cc:1428] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20250902 01:56:48.162168 30111 tablet_bootstrap.cc:654] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Neither blocks nor log segments found. Creating new log.
I20250902 01:56:48.170421 30111 tablet_bootstrap.cc:492] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: No bootstrap required, opened a new log
I20250902 01:56:48.170804 30111 ts_tablet_manager.cc:1397] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20250902 01:56:48.172683 30111 raft_consensus.cc:357] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.173053 30111 raft_consensus.cc:383] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:56:48.173223 30111 raft_consensus.cc:738] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Initialized, Role: FOLLOWER
I20250902 01:56:48.173696 30111 consensus_queue.cc:260] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.175048 30111 ts_tablet_manager.cc:1428] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250902 01:56:48.212651 30115 raft_consensus.cc:491] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.213004 30115 raft_consensus.cc:513] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.214356 30115 leader_election.cc:290] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:48.215211 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:48.215118 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.215693 29992 raft_consensus.cc:2466] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 0.
I20250902 01:56:48.215764 29918 raft_consensus.cc:2466] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 0.
I20250902 01:56:48.216450 30032 leader_election.cc:304] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.217000 30115 raft_consensus.cc:2802] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:48.217363 30115 raft_consensus.cc:491] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:48.217648 30115 raft_consensus.cc:3058] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.221413 30115 raft_consensus.cc:513] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.222707 30115 leader_election.cc:290] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:48.223364 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:48.223496 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74024619edd643b492526702db04fa94" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:48.223845 29918 raft_consensus.cc:3058] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.223937 29992 raft_consensus.cc:3058] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.227867 29992 raft_consensus.cc:2466] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 1.
I20250902 01:56:48.228650 30032 leader_election.cc:304] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.229189 30115 raft_consensus.cc:2802] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:48.229533 30115 raft_consensus.cc:695] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:56:48.229647 29918 raft_consensus.cc:2466] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 1.
I20250902 01:56:48.230530 30115 consensus_queue.cc:237] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.237982 29813 catalog_manager.cc:5582] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:56:48.290347 30147 consensus_queue.cc:1035] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.292307 30115 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.292697 30115 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.295301 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.295953 30115 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:48.296797 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:48.297463 29991 raft_consensus.cc:2391] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in current term 1: Already voted for candidate 33359a61000f449ab9b7172f83c3c946 in this term.
I20250902 01:56:48.299806 30032 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4; no voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.303022 30115 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.306578 30116 consensus_queue.cc:1035] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.309258 30115 raft_consensus.cc:2747] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250902 01:56:48.316345 29863 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:56:48.346942 30154 raft_consensus.cc:491] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.347338 30154 raft_consensus.cc:513] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:56:48.349071 30154 leader_election.cc:290] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:48.349749 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.350064 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:48.350696 29991 raft_consensus.cc:2391] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in current term 1: Already voted for candidate 33359a61000f449ab9b7172f83c3c946 in this term.
I20250902 01:56:48.351372 30032 leader_election.cc:304] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4; no voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.351918 30154 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.355787 30154 raft_consensus.cc:2747] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:48.375380 29918 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.380895 30120 consensus_queue.cc:1035] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.389837 30120 consensus_queue.cc:1035] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.416657 30116 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.417812 30154 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.418251 30154 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.420497 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.421154 29918 raft_consensus.cc:2391] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in current term 1: Already voted for candidate 968ef8b8113743b7a82cb596076f6c11 in this term.
I20250902 01:56:48.421370 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:48.422660 30032 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4; no voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.423334 30115 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.421154 30154 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:48.430146 29991 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.431078 30143 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.435927 30115 raft_consensus.cc:2747] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:56:48.445430 30144 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.452258 30120 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.469465 30115 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.480957 30115 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.482661 30147 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.485639 30147 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.496961 30145 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.505287 30144 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.567485 30120 raft_consensus.cc:491] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.567847 30147 raft_consensus.cc:491] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:56:48.568044 30120 raft_consensus.cc:513] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.568356 30147 raft_consensus.cc:513] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.569520 30120 leader_election.cc:290] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.570050 30147 leader_election.cc:290] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.570359 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946" is_pre_election: true
I20250902 01:56:48.570838 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:48.571334 30067 raft_consensus.cc:2466] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 0.
I20250902 01:56:48.570863 29918 raft_consensus.cc:2466] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 0.
I20250902 01:56:48.571947 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" is_pre_election: true
I20250902 01:56:48.571301 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11" is_pre_election: true
I20250902 01:56:48.572403 29957 leader_election.cc:304] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.572537 29991 raft_consensus.cc:2466] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:48.573127 30120 raft_consensus.cc:2802] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:48.573417 30120 raft_consensus.cc:491] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:48.573484 29883 leader_election.cc:304] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.573720 30120 raft_consensus.cc:3058] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.574190 30147 raft_consensus.cc:2802] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:56:48.574499 30147 raft_consensus.cc:491] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:56:48.574783 30147 raft_consensus.cc:3058] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.572484 30066 raft_consensus.cc:2466] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 0.
I20250902 01:56:48.579490 30120 raft_consensus.cc:513] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.579847 30147 raft_consensus.cc:513] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.581627 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:48.581805 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:48.582549 30147 leader_election.cc:290] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.582710 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:48.582259 30066 raft_consensus.cc:3058] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:56:48.582315 30120 leader_election.cc:290] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.582253 29918 raft_consensus.cc:2391] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 968ef8b8113743b7a82cb596076f6c11 in current term 1: Already voted for candidate 33359a61000f449ab9b7172f83c3c946 in this term.
I20250902 01:56:48.584918 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f17b201ff2784ad08192abc643e592ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:48.585570 29991 raft_consensus.cc:2391] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 33359a61000f449ab9b7172f83c3c946 in current term 1: Already voted for candidate 968ef8b8113743b7a82cb596076f6c11 in this term.
I20250902 01:56:48.586388 29883 leader_election.cc:304] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946; no voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.587040 30147 raft_consensus.cc:2747] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250902 01:56:48.588230 30066 raft_consensus.cc:2466] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 1.
I20250902 01:56:48.589043 29957 leader_election.cc:304] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters: 33359a61000f449ab9b7172f83c3c946
I20250902 01:56:48.589722 30120 raft_consensus.cc:2802] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:56:48.590126 30120 raft_consensus.cc:695] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:48.590724 30120 consensus_queue.cc:237] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.597765 29812 catalog_manager.cc:5582] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.620890 30187 consensus_queue.cc:1035] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.630429 30154 consensus_queue.cc:1035] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:48.798591 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:54530
I20250902 01:56:48.799227 29991 raft_consensus.cc:604] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Received request to transfer leadership to TS 33359a61000f449ab9b7172f83c3c946
I20250902 01:56:48.811132 30066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:56854
I20250902 01:56:48.811569 30066 raft_consensus.cc:604] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.823185 29918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:60090
I20250902 01:56:48.823589 29918 raft_consensus.cc:604] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:56:48.857805 30115 raft_consensus.cc:991] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:56:48.858299 30154 raft_consensus.cc:1079] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:56:48.859822 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54516
I20250902 01:56:48.860265 29991 raft_consensus.cc:491] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:56:48.860504 29991 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.864166 29991 raft_consensus.cc:513] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.865485 29991 leader_election.cc:290] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.866284 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:48.866482 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:48.866766 29918 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.867030 30066 raft_consensus.cc:3053] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:56:48.867324 30066 raft_consensus.cc:738] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:56:48.868099 30066 consensus_queue.cc:260] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.869341 30066 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.872964 29918 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:56:48.873370 30066 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:56:48.873914 29958 leader_election.cc:304] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.874619 30120 raft_consensus.cc:2802] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:56:48.874941 30120 raft_consensus.cc:695] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:48.875643 30120 consensus_queue.cc:237] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.880726 29812 catalog_manager.cc:5582] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 1 to 2, leader changed from 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:48.914033 30152 raft_consensus.cc:991] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:56:48.914366 30120 raft_consensus.cc:1079] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Signalling peer 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:56:48.915601 29918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:60076
I20250902 01:56:48.916041 29918 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:56:48.916419 29918 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.920692 29918 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.922000 29918 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 2 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:48.922631 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:48.923077 29991 raft_consensus.cc:3053] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:56:48.923321 29991 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:48.923846 29991 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.924638 29991 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.928448 29991 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 2.
I20250902 01:56:48.922948 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:48.928936 30066 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:48.929351 29883 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:48.930162 30147 raft_consensus.cc:2802] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:56:48.930568 30147 raft_consensus.cc:695] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:48.931216 30147 consensus_queue.cc:237] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:48.933153 30066 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 2.
I20250902 01:56:48.938068 29812 catalog_manager.cc:5582] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 1 to 2, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:49.059826 30116 raft_consensus.cc:991] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:56:49.060262 30147 raft_consensus.cc:1079] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:56:49.061389 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54500
I20250902 01:56:49.061794 29991 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:56:49.062057 29991 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:49.066195 29991 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:49.067581 29991 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:56:49.068256 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:49.068691 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:56:49.068837 29918 raft_consensus.cc:3053] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:56:49.069067 30066 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:49.069137 29918 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:49.069864 29918 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:49.070959 29918 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:56:49.073094 30066 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:56:49.073798 29957 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:49.074311 30120 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:56:49.074630 30120 raft_consensus.cc:695] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:49.075314 30120 consensus_queue.cc:237] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:49.076474 29918 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:56:49.081187 29812 catalog_manager.cc:5582] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 1 to 2, leader changed from 33359a61000f449ab9b7172f83c3c946 (127.29.21.193) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:56:49.170835 30120 consensus_queue.cc:1035] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:49.180058 30143 consensus_queue.cc:1035] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:56:49.312880 30066 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.313925 30147 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:49.320497 29991 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.322083 30116 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:49.382997 30066 raft_consensus.cc:1273] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.383911 30120 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:49.391126 29918 raft_consensus.cc:1273] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.392643 30120 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:49.473768 30066 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.474929 30120 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:49.481256 29918 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:56:49.482671 30143 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:56:51.855211 29783 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250902 01:56:51.855881 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:51.877331 29992 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:56:51.877744 29992 raft_consensus.cc:604] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Received request to transfer leadership to TS 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:51.886485 29918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
mode: GRACEFUL
new_leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:60100
I20250902 01:56:51.886886 29918 raft_consensus.cc:604] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 LEADER]: Received request to transfer leadership to TS 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:56:51.887790 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250902 01:56:51.888209 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:51.952582 30143 raft_consensus.cc:991] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:56:51.952936 30212 raft_consensus.cc:1079] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Signalling peer 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:56:51.953989 30066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:56846
I20250902 01:56:51.954391 30066 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:56:51.954636 30066 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:51.958511 30066 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:51.959760 30066 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:51.960484 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:51.960703 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:51.960882 29918 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:51.961160 29992 raft_consensus.cc:3053] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:56:51.961380 29992 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:56:51.961917 29992 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:51.962972 29992 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:51.964715 29918 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:56:51.965560 30032 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:56:51.966174 30205 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:56:51.967264 30205 raft_consensus.cc:695] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:56:51.968014 30205 consensus_queue.cc:237] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:51.968623 29992 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:56:51.975529 29813 catalog_manager.cc:5582] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 2 to 3, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:56:52.207945 30207 raft_consensus.cc:991] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: : Instructing follower 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:56:52.208338 30219 raft_consensus.cc:1079] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 LEADER]: Signalling peer 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:56:52.209604 30066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:56838
I20250902 01:56:52.210000 30066 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:56:52.210227 30066 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:52.213604 30066 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:52.214879 30066 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:56:52.215728 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:56:52.215881 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:56:52.216259 29918 raft_consensus.cc:3053] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:56:52.216316 29992 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:52.216648 29918 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:56:52.217326 29918 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:52.218395 29918 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:56:52.222420 29992 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:56:52.223313 30032 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:56:52.224002 30205 raft_consensus.cc:2802] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:56:52.224308 29918 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:56:52.224321 30205 raft_consensus.cc:695] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:56:52.225195 30205 consensus_queue.cc:237] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:56:52.231459 29813 catalog_manager.cc:5582] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 2 to 3, leader changed from 33359a61000f449ab9b7172f83c3c946 (127.29.21.193) to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:56:52.355822 29992 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:56:52.356806 30205 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:56:52.371577 29918 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:56:52.372870 30213 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:56:52.615005 29918 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:56:52.615998 30213 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:56:52.622968 29992 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:56:52.624153 30213 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:56:52.888973 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:52.893626 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:52.893934 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:53.894747 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:53.899140 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:53.899492 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:54.900231 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:54.904496 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:54.904832 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:55.905735 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:55.910272 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:55.910575 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:56.911378 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:56.915467 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:56.915804 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:57.916718 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:57.921067 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:57.921384 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:58.922363 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:58.927345 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:58.927752 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:56:59.928599 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:56:59.933488 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:56:59.933818 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:00.934523 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:00.939167 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:00.939529 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:01.962107 29783 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250902 01:57:01.965566 29918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:60090
I20250902 01:57:01.966137 29918 raft_consensus.cc:604] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:57:01.970009 29918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3344310a48194896aacca0abcdee4e52"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:60090
I20250902 01:57:01.970438 29918 raft_consensus.cc:604] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:57:01.974110 29918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:60090
I20250902 01:57:01.974459 29918 raft_consensus.cc:604] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:57:01.978181 30067 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:56854
I20250902 01:57:01.978564 30067 raft_consensus.cc:604] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:57:01.984117 30067 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
mode: GRACEFUL
new_leader_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:56854
I20250902 01:57:01.984521 30067 raft_consensus.cc:604] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Received request to transfer leadership to TS 968ef8b8113743b7a82cb596076f6c11
I20250902 01:57:01.994347 30235 raft_consensus.cc:991] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:01.994796 30239 raft_consensus.cc:1079] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:01.996238 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54500
I20250902 01:57:01.996590 29991 raft_consensus.cc:491] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:01.996838 29991 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.001224 29991 raft_consensus.cc:513] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:02.002601 29991 leader_election.cc:290] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:02.003400 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:02.003801 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:02.004123 29918 raft_consensus.cc:3053] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:02.004364 30067 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.004437 29918 raft_consensus.cc:738] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:02.005131 29918 consensus_queue.cc:260] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:02.006234 29918 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.101014 29918 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.101052 30067 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.102169 29958 leader_election.cc:304] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:57:02.102911 30231 raft_consensus.cc:2802] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:02.103817 30231 raft_consensus.cc:695] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:02.104710 30231 consensus_queue.cc:237] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:02.110785 29813 catalog_manager.cc:5582] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 1 to 2, leader changed from 33359a61000f449ab9b7172f83c3c946 (127.29.21.193) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:02.335860 30233 raft_consensus.cc:991] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.336287 30238 raft_consensus.cc:1079] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.337378 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54516
I20250902 01:57:02.337734 29991 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:02.337953 29991 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.341771 29991 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.343051 29991 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 4 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:02.343703 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:02.343950 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:02.344069 29918 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.344377 30067 raft_consensus.cc:3053] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:57:02.344607 30067 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:57:02.345173 30067 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.345989 30067 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.347841 29918 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 4.
I20250902 01:57:02.348693 29958 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:57:02.349336 30231 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:57:02.349776 30231 raft_consensus.cc:695] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:02.350382 30231 consensus_queue.cc:237] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.350724 30067 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 4.
I20250902 01:57:02.356310 29813 catalog_manager.cc:5582] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 3 to 4, leader changed from 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:02.385979 30233 raft_consensus.cc:991] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.386343 30233 raft_consensus.cc:1079] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.387513 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54516
I20250902 01:57:02.387900 29991 raft_consensus.cc:491] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:02.388152 29991 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.391886 29991 raft_consensus.cc:513] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.393056 29991 leader_election.cc:290] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 4 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:02.393590 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:02.393853 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "746252cfaffb40d0b9f70ad6a0b4a56c" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:02.394088 29918 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.394336 30067 raft_consensus.cc:3053] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:57:02.394615 30067 raft_consensus.cc:738] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:57:02.395267 30067 consensus_queue.cc:260] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.396246 30067 raft_consensus.cc:3058] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:02.399348 29918 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 4.
I20250902 01:57:02.400208 29958 leader_election.cc:304] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33359a61000f449ab9b7172f83c3c946, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:57:02.400753 30231 raft_consensus.cc:2802] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:57:02.401198 30231 raft_consensus.cc:695] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:02.401790 30067 raft_consensus.cc:2466] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 4.
I20250902 01:57:02.402019 30231 consensus_queue.cc:237] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.408547 29813 catalog_manager.cc:5582] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 3 to 4, leader changed from 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:02.461793 30235 raft_consensus.cc:991] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.462165 30239 raft_consensus.cc:1079] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.463171 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3344310a48194896aacca0abcdee4e52"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54500
I20250902 01:57:02.463519 29991 raft_consensus.cc:491] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:02.463801 29991 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.467257 29991 raft_consensus.cc:513] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.468434 29991 leader_election.cc:290] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:02.469125 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:02.469219 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3344310a48194896aacca0abcdee4e52" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:02.469533 29918 raft_consensus.cc:3053] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:02.469683 30067 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.469777 29918 raft_consensus.cc:738] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:02.470400 29918 consensus_queue.cc:260] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.471227 29918 raft_consensus.cc:3058] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.473912 30067 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.474747 29918 raft_consensus.cc:2466] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.474819 29957 leader_election.cc:304] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:57:02.475451 30231 raft_consensus.cc:2802] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:02.475821 30231 raft_consensus.cc:695] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:02.476454 30231 consensus_queue.cc:237] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.481634 29813 catalog_manager.cc:5582] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 1 to 2, leader changed from 33359a61000f449ab9b7172f83c3c946 (127.29.21.193) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:02.509971 30235 raft_consensus.cc:991] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: : Instructing follower 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.510352 30239 raft_consensus.cc:1079] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Signalling peer 968ef8b8113743b7a82cb596076f6c11 to start an election
I20250902 01:57:02.511469 29991 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
from {username='slave'} at 127.0.0.1:54500
I20250902 01:57:02.511883 29991 raft_consensus.cc:491] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:02.512135 29991 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.515470 29991 raft_consensus.cc:513] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.516585 29991 leader_election.cc:290] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:02.517189 29918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:02.517459 30067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "968ef8b8113743b7a82cb596076f6c11" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:02.517627 29918 raft_consensus.cc:3053] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:02.517900 29918 raft_consensus.cc:738] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:02.517951 30067 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.518496 29918 consensus_queue.cc:260] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.519258 29918 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:02.521886 30067 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.522581 29918 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968ef8b8113743b7a82cb596076f6c11 in term 2.
I20250902 01:57:02.522640 29957 leader_election.cc:304] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 968ef8b8113743b7a82cb596076f6c11; no voters:
I20250902 01:57:02.523252 30231 raft_consensus.cc:2802] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:02.523524 30231 raft_consensus.cc:695] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Leader. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:02.524070 30231 consensus_queue.cc:237] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:02.529620 29813 catalog_manager.cc:5582] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 reported cstate change: term changed from 1 to 2, leader changed from 33359a61000f449ab9b7172f83c3c946 (127.29.21.193) to 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "968ef8b8113743b7a82cb596076f6c11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:02.536482 30067 raft_consensus.cc:1273] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:02.537387 30231 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:02.553718 29918 raft_consensus.cc:1273] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:02.554586 30231 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:02.755331 30066 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:02.756340 30251 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:02.763645 29918 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:02.765131 30231 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:02.987303 30066 raft_consensus.cc:1273] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:02.988315 30231 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:02.995339 29918 raft_consensus.cc:1273] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:02.996591 30231 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:03.003423 30066 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:03.004266 30231 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:03.010954 29918 raft_consensus.cc:1273] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:03.012414 30231 consensus_queue.cc:1035] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:03.060328 29918 raft_consensus.cc:1273] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:03.061311 30251 consensus_queue.cc:1035] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:03.068173 30066 raft_consensus.cc:1273] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Refusing update from remote peer 968ef8b8113743b7a82cb596076f6c11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:03.069258 30231 consensus_queue.cc:1035] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:05.032418 29783 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250902 01:57:05.033169 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:05.039165 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:57:05.039798 29991 raft_consensus.cc:604] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Received request to transfer leadership to TS 33359a61000f449ab9b7172f83c3c946
I20250902 01:57:05.041754 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:57:05.042129 29991 raft_consensus.cc:604] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Received request to transfer leadership to TS 33359a61000f449ab9b7172f83c3c946
I20250902 01:57:05.043771 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:57:05.044142 29991 raft_consensus.cc:604] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Received request to transfer leadership to TS 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:57:05.045706 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:57:05.046080 29991 raft_consensus.cc:604] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Received request to transfer leadership to TS 33359a61000f449ab9b7172f83c3c946
I20250902 01:57:05.047611 29991 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5"
dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
mode: GRACEFUL
new_leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:54542
I20250902 01:57:05.047930 29991 raft_consensus.cc:604] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Received request to transfer leadership to TS 0e43fcdd1b4540189cdd1af2bfdca2d4
I20250902 01:57:05.048655 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250902 01:57:05.048959 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:05.074385 30240 raft_consensus.cc:991] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.074853 30265 raft_consensus.cc:1079] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Signalling peer 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.076066 29917 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:60076
I20250902 01:57:05.076440 29917 raft_consensus.cc:491] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:05.076727 29917 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.082034 29917 raft_consensus.cc:513] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.084204 29917 leader_election.cc:290] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 3 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:05.085611 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:05.085368 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "184c16e4c06d492eba1c49f5ccad2d88" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:57:05.086102 30066 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.086226 29991 raft_consensus.cc:3053] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:05.086618 29991 raft_consensus.cc:738] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:05.087400 29991 consensus_queue.cc:260] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.088574 29991 raft_consensus.cc:3058] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.091372 30066 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 3.
I20250902 01:57:05.092371 29882 leader_election.cc:304] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:57:05.093183 29991 raft_consensus.cc:2466] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 3.
I20250902 01:57:05.093808 30267 raft_consensus.cc:2802] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:05.094645 30267 raft_consensus.cc:695] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 3 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:05.095360 30267 consensus_queue.cc:237] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.101939 29812 catalog_manager.cc:5582] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 2 to 3, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:05.162191 30240 raft_consensus.cc:991] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:57:05.162585 30265 raft_consensus.cc:1079] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Signalling peer 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:57:05.163731 30066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:56846
I20250902 01:57:05.164103 30066 raft_consensus.cc:491] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:05.164321 30066 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:05.167897 30066 raft_consensus.cc:513] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.169494 30066 leader_election.cc:290] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 2 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:57:05.170395 29917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:05.170439 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53a0165266f441bdaba7be5d35c7e58a" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:57:05.170945 29917 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:05.171005 29991 raft_consensus.cc:3053] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:05.171291 29991 raft_consensus.cc:738] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:05.171801 29991 consensus_queue.cc:260] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.172552 29991 raft_consensus.cc:3058] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:05.175061 29917 raft_consensus.cc:2466] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 2.
I20250902 01:57:05.176064 29991 raft_consensus.cc:2466] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 2.
I20250902 01:57:05.176062 30032 leader_election.cc:304] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:57:05.176924 30233 raft_consensus.cc:2802] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:05.177767 30233 raft_consensus.cc:695] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:57:05.178382 30233 consensus_queue.cc:237] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.184623 29812 catalog_manager.cc:5582] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 1 to 2, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:05.228747 30240 raft_consensus.cc:991] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.229115 30265 raft_consensus.cc:1079] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Signalling peer 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.230286 29917 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:60076
I20250902 01:57:05.230679 29917 raft_consensus.cc:491] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:05.230942 29917 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.234930 29917 raft_consensus.cc:513] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:05.236199 29917 leader_election.cc:290] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 3 election: Requested vote from peers 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:57:05.236814 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:05.237007 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1512fda84c5a470fbcf5b6080b4ab3ff" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:57:05.237341 30066 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.237392 29991 raft_consensus.cc:3053] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:05.237707 29991 raft_consensus.cc:738] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:05.238394 29991 consensus_queue.cc:260] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:05.239558 29991 raft_consensus.cc:3058] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.242847 30066 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 3.
I20250902 01:57:05.243683 29882 leader_election.cc:304] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:57:05.244210 30267 raft_consensus.cc:2802] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:05.244618 30267 raft_consensus.cc:695] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 3 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:05.244669 29991 raft_consensus.cc:2466] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 3.
I20250902 01:57:05.245464 30267 consensus_queue.cc:237] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } }
I20250902 01:57:05.257936 29812 catalog_manager.cc:5582] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 2 to 3, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:05.387826 30240 raft_consensus.cc:991] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.388195 30265 raft_consensus.cc:1079] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Signalling peer 33359a61000f449ab9b7172f83c3c946 to start an election
I20250902 01:57:05.389276 29917 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "7375387eca6f4ec1886e409e60131330"
dest_uuid: "33359a61000f449ab9b7172f83c3c946"
from {username='slave'} at 127.0.0.1:60076
I20250902 01:57:05.389701 29917 raft_consensus.cc:491] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:05.389933 29917 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:05.393983 29917 raft_consensus.cc:513] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.395169 29917 leader_election.cc:290] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 5 election: Requested vote from peers 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815), 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195:36599)
I20250902 01:57:05.395825 29991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:57:05.396001 30066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7375387eca6f4ec1886e409e60131330" candidate_uuid: "33359a61000f449ab9b7172f83c3c946" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
I20250902 01:57:05.396205 29991 raft_consensus.cc:3053] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:57:05.396443 30066 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:05.396463 29991 raft_consensus.cc:738] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:05.397083 29991 consensus_queue.cc:260] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.397889 29991 raft_consensus.cc:3058] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:05.400090 30066 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 5.
I20250902 01:57:05.400766 29882 leader_election.cc:304] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:57:05.401240 30267 raft_consensus.cc:2802] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:57:05.401588 30267 raft_consensus.cc:695] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 5 LEADER]: Becoming Leader. State: Replica: 33359a61000f449ab9b7172f83c3c946, State: Running, Role: LEADER
I20250902 01:57:05.401683 29991 raft_consensus.cc:2466] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33359a61000f449ab9b7172f83c3c946 in term 5.
I20250902 01:57:05.402328 30267 consensus_queue.cc:237] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.407617 29812 catalog_manager.cc:5582] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 reported cstate change: term changed from 4 to 5, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 33359a61000f449ab9b7172f83c3c946 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "33359a61000f449ab9b7172f83c3c946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:05.561563 30240 raft_consensus.cc:991] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: : Instructing follower 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:57:05.561977 30265 raft_consensus.cc:1079] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Signalling peer 0e43fcdd1b4540189cdd1af2bfdca2d4 to start an election
I20250902 01:57:05.563105 30066 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5"
dest_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4"
from {username='slave'} at 127.0.0.1:56846
I20250902 01:57:05.563443 30066 raft_consensus.cc:491] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:05.563659 30066 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.565241 29991 raft_consensus.cc:1273] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:05.566104 30233 consensus_queue.cc:1035] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:05.568398 30066 raft_consensus.cc:513] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.569723 30066 leader_election.cc:290] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Requested vote from peers 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657), 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194:33815)
I20250902 01:57:05.570676 29917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "33359a61000f449ab9b7172f83c3c946"
I20250902 01:57:05.571326 29917 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.571679 29992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd26046f3ae04db7ae1542964786ceb5" candidate_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "968ef8b8113743b7a82cb596076f6c11"
I20250902 01:57:05.572193 29992 raft_consensus.cc:3053] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:05.572415 29992 raft_consensus.cc:738] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 968ef8b8113743b7a82cb596076f6c11, State: Running, Role: LEADER
I20250902 01:57:05.572994 29992 consensus_queue.cc:260] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.573815 29992 raft_consensus.cc:3058] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:05.577947 29917 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:57:05.579041 30032 leader_election.cc:304] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e43fcdd1b4540189cdd1af2bfdca2d4, 33359a61000f449ab9b7172f83c3c946; no voters:
I20250902 01:57:05.579386 29992 raft_consensus.cc:2466] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e43fcdd1b4540189cdd1af2bfdca2d4 in term 3.
I20250902 01:57:05.580363 30274 raft_consensus.cc:2802] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:05.580982 29917 raft_consensus.cc:1273] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:05.589268 30274 raft_consensus.cc:695] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Becoming Leader. State: Replica: 0e43fcdd1b4540189cdd1af2bfdca2d4, State: Running, Role: LEADER
I20250902 01:57:05.590001 30282 consensus_queue.cc:1035] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:57:05.592545 30274 consensus_queue.cc:237] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } }
I20250902 01:57:05.601276 29812 catalog_manager.cc:5582] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 reported cstate change: term changed from 2 to 3, leader changed from 968ef8b8113743b7a82cb596076f6c11 (127.29.21.194) to 0e43fcdd1b4540189cdd1af2bfdca2d4 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:05.635038 30066 raft_consensus.cc:1273] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:05.635949 30267 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:05.643141 29991 raft_consensus.cc:1273] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:05.644678 30267 consensus_queue.cc:1035] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:05.677429 29991 raft_consensus.cc:1273] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:05.678267 30267 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:05.685375 30066 raft_consensus.cc:1273] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:05.687322 30288 consensus_queue.cc:1035] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:05.841492 29991 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 5 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250902 01:57:05.842500 30288 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:57:05.849946 30066 raft_consensus.cc:1273] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 5 FOLLOWER]: Refusing update from remote peer 33359a61000f449ab9b7172f83c3c946: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250902 01:57:05.851519 30288 consensus_queue.cc:1035] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e43fcdd1b4540189cdd1af2bfdca2d4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:57:06.000404 29991 raft_consensus.cc:1273] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:06.001257 30274 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968ef8b8113743b7a82cb596076f6c11" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:06.008304 29917 raft_consensus.cc:1273] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Refusing update from remote peer 0e43fcdd1b4540189cdd1af2bfdca2d4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:06.009897 30274 consensus_queue.cc:1035] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33359a61000f449ab9b7172f83c3c946" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:06.049872 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:06.054209 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:06.054524 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:07.055393 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:07.060104 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:07.060428 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:08.061296 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:08.065739 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:08.066107 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:09.066943 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:09.071578 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:09.071926 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:10.072819 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:10.077194 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:10.077553 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:11.078521 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:11.083420 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:11.083814 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:12.084801 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:12.089731 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:12.090081 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:13.091045 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:13.096385 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:13.096735 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:14.097695 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:14.102154 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:14.102490 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:15.125963 29783 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250902 01:57:15.126736 29783 tablet_server.cc:178] TabletServer@127.29.21.193:0 shutting down...
I20250902 01:57:15.151371 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:15.152089 29783 tablet_replica.cc:331] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.152778 29783 raft_consensus.cc:2241] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.153316 29783 raft_consensus.cc:2270] T f17b201ff2784ad08192abc643e592ff P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.155417 29783 tablet_replica.cc:331] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.155921 29783 raft_consensus.cc:2241] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.156220 29783 raft_consensus.cc:2270] T 53a0165266f441bdaba7be5d35c7e58a P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.157670 29783 tablet_replica.cc:331] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.158104 29783 raft_consensus.cc:2241] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.158437 29783 raft_consensus.cc:2270] T 74024619edd643b492526702db04fa94 P 33359a61000f449ab9b7172f83c3c946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.159873 29783 tablet_replica.cc:331] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.160285 29783 raft_consensus.cc:2241] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 3 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.161000 29783 raft_consensus.cc:2270] T 1512fda84c5a470fbcf5b6080b4ab3ff P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.162390 29783 tablet_replica.cc:331] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.162815 29783 raft_consensus.cc:2241] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 5 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.163519 29783 raft_consensus.cc:2270] T 7375387eca6f4ec1886e409e60131330 P 33359a61000f449ab9b7172f83c3c946 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.164920 29783 tablet_replica.cc:331] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.165284 29783 raft_consensus.cc:2241] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.165617 29783 raft_consensus.cc:2270] T bd26046f3ae04db7ae1542964786ceb5 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.166935 29783 tablet_replica.cc:331] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.167284 29783 raft_consensus.cc:2241] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 3 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.168033 29783 raft_consensus.cc:2270] T 184c16e4c06d492eba1c49f5ccad2d88 P 33359a61000f449ab9b7172f83c3c946 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.169401 29783 tablet_replica.cc:331] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.169765 29783 raft_consensus.cc:2241] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.170218 29783 raft_consensus.cc:2270] T 3344310a48194896aacca0abcdee4e52 P 33359a61000f449ab9b7172f83c3c946 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.171614 29783 tablet_replica.cc:331] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946: stopping tablet replica
I20250902 01:57:15.171969 29783 raft_consensus.cc:2241] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.172310 29783 raft_consensus.cc:2270] T 746252cfaffb40d0b9f70ad6a0b4a56c P 33359a61000f449ab9b7172f83c3c946 [term 4 FOLLOWER]: Raft consensus is shut down!
W20250902 01:57:15.281176 30032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.21.193:42657: connect: Connection refused (error 111)
W20250902 01:57:15.281179 29958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.21.193:42657: connect: Connection refused (error 111)
I20250902 01:57:15.284890 29783 tablet_server.cc:195] TabletServer@127.29.21.193:0 shutdown complete.
W20250902 01:57:15.285326 29958 consensus_peers.cc:489] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 -> Peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Couldn't send request to peer 33359a61000f449ab9b7172f83c3c946. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:42657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:15.286401 30032 consensus_peers.cc:489] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 -> Peer 33359a61000f449ab9b7172f83c3c946 (127.29.21.193:42657): Couldn't send request to peer 33359a61000f449ab9b7172f83c3c946. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:42657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:15.302809 29783 tablet_server.cc:178] TabletServer@127.29.21.194:0 shutting down...
I20250902 01:57:15.329159 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:15.329689 29783 tablet_replica.cc:331] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.330225 29783 raft_consensus.cc:2241] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.330924 29783 raft_consensus.cc:2270] T f17b201ff2784ad08192abc643e592ff P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.332499 29783 tablet_replica.cc:331] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.332986 29783 raft_consensus.cc:2241] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.333407 29783 raft_consensus.cc:2270] T 184c16e4c06d492eba1c49f5ccad2d88 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.334856 29783 tablet_replica.cc:331] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.335254 29783 raft_consensus.cc:2241] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.335939 29783 raft_consensus.cc:2270] T 3344310a48194896aacca0abcdee4e52 P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.337564 29783 tablet_replica.cc:331] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.337996 29783 raft_consensus.cc:2241] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 4 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.338680 29783 raft_consensus.cc:2270] T 746252cfaffb40d0b9f70ad6a0b4a56c P 968ef8b8113743b7a82cb596076f6c11 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.340397 29783 tablet_replica.cc:331] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.340798 29783 raft_consensus.cc:2241] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.341128 29783 raft_consensus.cc:2270] T bd26046f3ae04db7ae1542964786ceb5 P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.342432 29783 tablet_replica.cc:331] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.342886 29783 raft_consensus.cc:2241] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.343243 29783 raft_consensus.cc:2270] T 7375387eca6f4ec1886e409e60131330 P 968ef8b8113743b7a82cb596076f6c11 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.344648 29783 tablet_replica.cc:331] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.345057 29783 raft_consensus.cc:2241] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.345388 29783 raft_consensus.cc:2270] T 1512fda84c5a470fbcf5b6080b4ab3ff P 968ef8b8113743b7a82cb596076f6c11 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.346855 29783 tablet_replica.cc:331] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.347242 29783 raft_consensus.cc:2241] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.347707 29783 raft_consensus.cc:2270] T 74024619edd643b492526702db04fa94 P 968ef8b8113743b7a82cb596076f6c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.349071 29783 tablet_replica.cc:331] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11: stopping tablet replica
I20250902 01:57:15.349453 29783 raft_consensus.cc:2241] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.349787 29783 raft_consensus.cc:2270] T 53a0165266f441bdaba7be5d35c7e58a P 968ef8b8113743b7a82cb596076f6c11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.374451 29783 tablet_server.cc:195] TabletServer@127.29.21.194:0 shutdown complete.
I20250902 01:57:15.390476 29783 tablet_server.cc:178] TabletServer@127.29.21.195:0 shutting down...
I20250902 01:57:15.410964 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:15.411688 29783 tablet_replica.cc:331] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.412160 29783 raft_consensus.cc:2241] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.412951 29783 raft_consensus.cc:2270] T 74024619edd643b492526702db04fa94 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.414530 29783 tablet_replica.cc:331] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.414942 29783 raft_consensus.cc:2241] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.415305 29783 raft_consensus.cc:2270] T 746252cfaffb40d0b9f70ad6a0b4a56c P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.416601 29783 tablet_replica.cc:331] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.416985 29783 raft_consensus.cc:2241] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.417268 29783 raft_consensus.cc:2270] T 3344310a48194896aacca0abcdee4e52 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.418677 29783 tablet_replica.cc:331] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.419060 29783 raft_consensus.cc:2241] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.419413 29783 raft_consensus.cc:2270] T 184c16e4c06d492eba1c49f5ccad2d88 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.420771 29783 tablet_replica.cc:331] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.421152 29783 raft_consensus.cc:2241] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.421437 29783 raft_consensus.cc:2270] T f17b201ff2784ad08192abc643e592ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.422784 29783 tablet_replica.cc:331] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.423168 29783 raft_consensus.cc:2241] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.423862 29783 raft_consensus.cc:2270] T 53a0165266f441bdaba7be5d35c7e58a P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.425069 29783 tablet_replica.cc:331] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.425442 29783 raft_consensus.cc:2241] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.425827 29783 raft_consensus.cc:2270] T 7375387eca6f4ec1886e409e60131330 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.427078 29783 tablet_replica.cc:331] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.427467 29783 raft_consensus.cc:2241] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:15.427875 29783 raft_consensus.cc:2270] T 1512fda84c5a470fbcf5b6080b4ab3ff P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.429177 29783 tablet_replica.cc:331] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4: stopping tablet replica
I20250902 01:57:15.429601 29783 raft_consensus.cc:2241] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.430200 29783 raft_consensus.cc:2270] T bd26046f3ae04db7ae1542964786ceb5 P 0e43fcdd1b4540189cdd1af2bfdca2d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.455989 29783 tablet_server.cc:195] TabletServer@127.29.21.195:0 shutdown complete.
I20250902 01:57:15.471236 29783 master.cc:561] Master@127.29.21.254:46173 shutting down...
I20250902 01:57:15.580269 29783 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 1 LEADER]: Raft consensus shutting down.
I20250902 01:57:15.580787 29783 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:15.581045 29783 tablet_replica.cc:331] T 00000000000000000000000000000000 P e0be53f1efc148e6a9b2a8b2d31bd2ac: stopping tablet replica
I20250902 01:57:15.599316 29783 master.cc:583] Master@127.29.21.254:46173 shutdown complete.
0e43fcdd1b4540189cdd1af2bfdca2d4 1
33359a61000f449ab9b7172f83c3c946 4
968ef8b8113743b7a82cb596076f6c11 4
0e43fcdd1b4540189cdd1af2bfdca2d4 3
33359a61000f449ab9b7172f83c3c946 3
968ef8b8113743b7a82cb596076f6c11 3
0e43fcdd1b4540189cdd1af2bfdca2d4 1
33359a61000f449ab9b7172f83c3c946 0
968ef8b8113743b7a82cb596076f6c11 8
0e43fcdd1b4540189cdd1af2bfdca2d4 3
33359a61000f449ab9b7172f83c3c946 3
968ef8b8113743b7a82cb596076f6c11 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30685 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250902 01:57:15.634778 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:43255
I20250902 01:57:15.635653 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:15.640209 30306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:15.640442 30307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:15.641615 30309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:15.642315 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:15.643117 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:15.643316 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:15.643450 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778235643433 us; error 0 us; skew 500 ppm
I20250902 01:57:15.643988 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:15.645829 29783 webserver.cc:480] Webserver started at http://127.29.21.254:33519/ using document root <none> and password file <none>
I20250902 01:57:15.646231 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:15.646405 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:15.646617 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:15.647708 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/master-0-root/instance:
uuid: "87409d5661214b34a0d19f6b74641dc4"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.651484 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250902 01:57:15.654433 30314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.655079 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250902 01:57:15.655282 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/master-0-root
uuid: "87409d5661214b34a0d19f6b74641dc4"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.655483 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:15.690240 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:15.691287 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:15.735491 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:43255
I20250902 01:57:15.735589 30365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.254:43255 every 8 connection(s)
I20250902 01:57:15.738798 30366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:15.747138 30366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4: Bootstrap starting.
I20250902 01:57:15.751262 30366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:15.754885 30366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4: No bootstrap required, opened a new log
I20250902 01:57:15.756896 30366 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER }
I20250902 01:57:15.757301 30366 raft_consensus.cc:383] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:15.757491 30366 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87409d5661214b34a0d19f6b74641dc4, State: Initialized, Role: FOLLOWER
I20250902 01:57:15.757881 30366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER }
I20250902 01:57:15.758244 30366 raft_consensus.cc:397] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250902 01:57:15.758423 30366 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250902 01:57:15.758622 30366 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:15.761940 30366 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER }
I20250902 01:57:15.762359 30366 leader_election.cc:304] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87409d5661214b34a0d19f6b74641dc4; no voters:
I20250902 01:57:15.763362 30366 leader_election.cc:290] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250902 01:57:15.763731 30369 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:15.764855 30369 raft_consensus.cc:695] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 1 LEADER]: Becoming Leader. State: Replica: 87409d5661214b34a0d19f6b74641dc4, State: Running, Role: LEADER
I20250902 01:57:15.765358 30369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER }
I20250902 01:57:15.766022 30366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [sys.catalog]: configured and running, proceeding with master startup.
I20250902 01:57:15.768061 30370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87409d5661214b34a0d19f6b74641dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER } }
I20250902 01:57:15.768136 30371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87409d5661214b34a0d19f6b74641dc4. Latest consensus state: current_term: 1 leader_uuid: "87409d5661214b34a0d19f6b74641dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87409d5661214b34a0d19f6b74641dc4" member_type: VOTER } }
I20250902 01:57:15.768791 30370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [sys.catalog]: This master's current role is: LEADER
I20250902 01:57:15.768838 30371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [sys.catalog]: This master's current role is: LEADER
I20250902 01:57:15.771139 30375 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250902 01:57:15.775854 30375 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250902 01:57:15.781016 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250902 01:57:15.783790 30375 catalog_manager.cc:1349] Generated new cluster ID: e2ec10a0159541fc8c054426e24f7786
I20250902 01:57:15.784005 30375 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250902 01:57:15.803674 30375 catalog_manager.cc:1372] Generated new certificate authority record
I20250902 01:57:15.804651 30375 catalog_manager.cc:1506] Loading token signing keys...
I20250902 01:57:15.814066 30375 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4: Generated new TSK 0
I20250902 01:57:15.814530 30375 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250902 01:57:15.847508 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:15.852931 30387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:15.854178 30388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:15.855578 29783 server_base.cc:1047] running on GCE node
W20250902 01:57:15.855983 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:15.856653 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:15.856822 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:15.856969 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778235856954 us; error 0 us; skew 500 ppm
I20250902 01:57:15.857448 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:15.859361 29783 webserver.cc:480] Webserver started at http://127.29.21.193:34863/ using document root <none> and password file <none>
I20250902 01:57:15.859838 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:15.860016 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:15.860255 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:15.861195 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-0-root/instance:
uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.865185 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250902 01:57:15.867991 30395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.868652 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250902 01:57:15.868902 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.869139 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:15.883684 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:15.884692 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:15.885968 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:15.887918 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:15.888092 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.888348 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:15.888514 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.926082 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:33507
I20250902 01:57:15.926163 30457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.193:33507 every 8 connection(s)
I20250902 01:57:15.930305 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:15.936509 30462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:15.937455 30463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:15.940968 30465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:15.941160 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:15.941756 30458 heartbeater.cc:344] Connected to a master server at 127.29.21.254:43255
I20250902 01:57:15.942116 30458 heartbeater.cc:461] Registering TS with master...
I20250902 01:57:15.942299 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:15.942584 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:15.942778 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778235942757 us; error 0 us; skew 500 ppm
I20250902 01:57:15.942925 30458 heartbeater.cc:507] Master 127.29.21.254:43255 requested a full tablet report, sending...
I20250902 01:57:15.943463 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:15.945060 30331 ts_manager.cc:194] Registered new tserver with Master: 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507)
I20250902 01:57:15.945951 29783 webserver.cc:480] Webserver started at http://127.29.21.194:36965/ using document root <none> and password file <none>
I20250902 01:57:15.946430 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:15.946604 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:15.946731 30331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52866
I20250902 01:57:15.946825 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:15.948141 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-1-root/instance:
uuid: "673abf5ce13140a7b854b7f50e0f6488"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.951850 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250902 01:57:15.954535 30470 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.955112 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250902 01:57:15.955346 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
uuid: "673abf5ce13140a7b854b7f50e0f6488"
format_stamp: "Formatted at 2025-09-02 01:57:15 on dist-test-slave-g0c9"
I20250902 01:57:15.955596 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:15.986866 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:15.987993 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:15.989320 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:15.991350 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:15.991518 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:15.991729 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:15.991864 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:16.026573 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:40563
I20250902 01:57:16.026661 30532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.194:40563 every 8 connection(s)
I20250902 01:57:16.030771 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:16.036983 30536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:16.037508 30537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:16.039705 29783 server_base.cc:1047] running on GCE node
W20250902 01:57:16.041034 30540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:16.041759 30533 heartbeater.cc:344] Connected to a master server at 127.29.21.254:43255
I20250902 01:57:16.041980 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250902 01:57:16.042047 30533 heartbeater.cc:461] Registering TS with master...
W20250902 01:57:16.042260 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:16.042454 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778236042436 us; error 0 us; skew 500 ppm
I20250902 01:57:16.042850 30533 heartbeater.cc:507] Master 127.29.21.254:43255 requested a full tablet report, sending...
I20250902 01:57:16.042976 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:16.044588 30331 ts_manager.cc:194] Registered new tserver with Master: 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.045166 29783 webserver.cc:480] Webserver started at http://127.29.21.195:33029/ using document root <none> and password file <none>
I20250902 01:57:16.045609 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:16.045773 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:16.045799 30331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52868
I20250902 01:57:16.046041 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:16.047222 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-2-root/instance:
uuid: "5437b7338b794400b09a7d407549227a"
format_stamp: "Formatted at 2025-09-02 01:57:16 on dist-test-slave-g0c9"
I20250902 01:57:16.051084 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250902 01:57:16.053992 30544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:16.054733 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250902 01:57:16.054953 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
uuid: "5437b7338b794400b09a7d407549227a"
format_stamp: "Formatted at 2025-09-02 01:57:16 on dist-test-slave-g0c9"
I20250902 01:57:16.055160 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756778204804806-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:16.065254 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:16.066218 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:16.067476 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:16.069501 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:16.069662 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:16.069824 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:16.069943 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:16.105129 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:40753
I20250902 01:57:16.105185 30606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.195:40753 every 8 connection(s)
I20250902 01:57:16.116932 30607 heartbeater.cc:344] Connected to a master server at 127.29.21.254:43255
I20250902 01:57:16.117223 30607 heartbeater.cc:461] Registering TS with master...
I20250902 01:57:16.117867 30607 heartbeater.cc:507] Master 127.29.21.254:43255 requested a full tablet report, sending...
I20250902 01:57:16.119359 30331 ts_manager.cc:194] Registered new tserver with Master: 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.120045 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012052973s
I20250902 01:57:16.120711 30331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52880
I20250902 01:57:16.122633 29783 test_util.cc:276] Using random seed: -826894857
I20250902 01:57:16.141099 30331 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52896:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250902 01:57:16.143721 30331 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250902 01:57:16.211892 30423 tablet_service.cc:1468] Processing CreateTablet for tablet ef2dd292e59f4e28b6349b96fdba1b04 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250902 01:57:16.213342 30423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2dd292e59f4e28b6349b96fdba1b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.216753 30572 tablet_service.cc:1468] Processing CreateTablet for tablet ef2dd292e59f4e28b6349b96fdba1b04 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250902 01:57:16.217970 30572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2dd292e59f4e28b6349b96fdba1b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.218576 30566 tablet_service.cc:1468] Processing CreateTablet for tablet c12b7b1fd6ef4af2a1a7e3917a553838 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250902 01:57:16.218627 30422 tablet_service.cc:1468] Processing CreateTablet for tablet 2746e7eebde8423f81678907aba6af92 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250902 01:57:16.219861 30566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12b7b1fd6ef4af2a1a7e3917a553838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.220628 30421 tablet_service.cc:1468] Processing CreateTablet for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250902 01:57:16.221881 30421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.221793 30570 tablet_service.cc:1468] Processing CreateTablet for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250902 01:57:16.223129 30570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.220640 30571 tablet_service.cc:1468] Processing CreateTablet for tablet 2746e7eebde8423f81678907aba6af92 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250902 01:57:16.224643 30571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2746e7eebde8423f81678907aba6af92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.225683 30568 tablet_service.cc:1468] Processing CreateTablet for tablet 30272acb192f4b15864059546a8019a3 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250902 01:57:16.229054 30567 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5844121c7e4b4c968a544c1b568e90 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250902 01:57:16.224658 30569 tablet_service.cc:1468] Processing CreateTablet for tablet 777e53e58e4242408e7377ffe8ecb5a8 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250902 01:57:16.230420 30422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2746e7eebde8423f81678907aba6af92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.226919 30568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30272acb192f4b15864059546a8019a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.235939 30569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777e53e58e4242408e7377ffe8ecb5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.236502 30565 tablet_service.cc:1468] Processing CreateTablet for tablet 43d6dabf0ff34265a47b8b39c2e84266 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250902 01:57:16.237274 30567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5844121c7e4b4c968a544c1b568e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.238478 30564 tablet_service.cc:1468] Processing CreateTablet for tablet eb10f324c60e4436851b4abbcf2a503d (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250902 01:57:16.240521 30563 tablet_service.cc:1468] Processing CreateTablet for tablet d404cc6e160a41458b944ce1811659cf (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250902 01:57:16.241528 30564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10f324c60e4436851b4abbcf2a503d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.241295 30420 tablet_service.cc:1468] Processing CreateTablet for tablet 777e53e58e4242408e7377ffe8ecb5a8 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250902 01:57:16.242023 30565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d6dabf0ff34265a47b8b39c2e84266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.242519 30420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777e53e58e4242408e7377ffe8ecb5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.242871 30563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d404cc6e160a41458b944ce1811659cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.244676 30419 tablet_service.cc:1468] Processing CreateTablet for tablet 30272acb192f4b15864059546a8019a3 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250902 01:57:16.245812 30419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30272acb192f4b15864059546a8019a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.248697 30418 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5844121c7e4b4c968a544c1b568e90 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250902 01:57:16.249859 30418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5844121c7e4b4c968a544c1b568e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.267400 30416 tablet_service.cc:1468] Processing CreateTablet for tablet 43d6dabf0ff34265a47b8b39c2e84266 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250902 01:57:16.267839 30415 tablet_service.cc:1468] Processing CreateTablet for tablet eb10f324c60e4436851b4abbcf2a503d (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250902 01:57:16.268757 30416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d6dabf0ff34265a47b8b39c2e84266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.267848 30417 tablet_service.cc:1468] Processing CreateTablet for tablet c12b7b1fd6ef4af2a1a7e3917a553838 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250902 01:57:16.270120 30417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12b7b1fd6ef4af2a1a7e3917a553838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.278661 30414 tablet_service.cc:1468] Processing CreateTablet for tablet d404cc6e160a41458b944ce1811659cf (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250902 01:57:16.279695 30415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10f324c60e4436851b4abbcf2a503d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.280560 30498 tablet_service.cc:1468] Processing CreateTablet for tablet ef2dd292e59f4e28b6349b96fdba1b04 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250902 01:57:16.284062 30492 tablet_service.cc:1468] Processing CreateTablet for tablet c12b7b1fd6ef4af2a1a7e3917a553838 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250902 01:57:16.285948 30495 tablet_service.cc:1468] Processing CreateTablet for tablet 777e53e58e4242408e7377ffe8ecb5a8 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250902 01:57:16.287906 30493 tablet_service.cc:1468] Processing CreateTablet for tablet 8d5844121c7e4b4c968a544c1b568e90 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250902 01:57:16.281795 30498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2dd292e59f4e28b6349b96fdba1b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.289814 30414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d404cc6e160a41458b944ce1811659cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.290400 30491 tablet_service.cc:1468] Processing CreateTablet for tablet 43d6dabf0ff34265a47b8b39c2e84266 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250902 01:57:16.292631 30490 tablet_service.cc:1468] Processing CreateTablet for tablet eb10f324c60e4436851b4abbcf2a503d (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250902 01:57:16.294711 30496 tablet_service.cc:1468] Processing CreateTablet for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250902 01:57:16.296671 30494 tablet_service.cc:1468] Processing CreateTablet for tablet 30272acb192f4b15864059546a8019a3 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250902 01:57:16.298765 30489 tablet_service.cc:1468] Processing CreateTablet for tablet d404cc6e160a41458b944ce1811659cf (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250902 01:57:16.301250 30491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43d6dabf0ff34265a47b8b39c2e84266. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.301395 30497 tablet_service.cc:1468] Processing CreateTablet for tablet 2746e7eebde8423f81678907aba6af92 (DEFAULT_TABLE table=test-workload [id=2239e47973c949d3b719aa718c4a20bb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250902 01:57:16.302597 30497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2746e7eebde8423f81678907aba6af92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.303836 30496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d12e1a2135c4ebeaf1adf6edb9aa07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.304409 30494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30272acb192f4b15864059546a8019a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.304961 30492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12b7b1fd6ef4af2a1a7e3917a553838. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.312255 30495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777e53e58e4242408e7377ffe8ecb5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.312922 30490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb10f324c60e4436851b4abbcf2a503d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.313500 30493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5844121c7e4b4c968a544c1b568e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.315824 30489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d404cc6e160a41458b944ce1811659cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:16.325371 30622 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.364624 30623 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.374485 30624 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.379572 30624 tablet_bootstrap.cc:654] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.386662 30622 tablet_bootstrap.cc:654] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.389125 30623 tablet_bootstrap.cc:654] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.392892 30624 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.393347 30624 ts_tablet_manager.cc:1397] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
I20250902 01:57:16.395846 30622 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.396155 30622 ts_tablet_manager.cc:1397] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.004s
I20250902 01:57:16.395720 30624 raft_consensus.cc:357] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.396390 30624 raft_consensus.cc:383] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.396541 30623 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.396667 30624 raft_consensus.cc:738] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.396911 30623 ts_tablet_manager.cc:1397] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.003s
I20250902 01:57:16.397275 30624 consensus_queue.cc:260] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.397794 30622 raft_consensus.cc:357] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.398217 30622 raft_consensus.cc:383] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.398430 30622 raft_consensus.cc:738] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.399178 30458 heartbeater.cc:499] Master 127.29.21.254:43255 was elected leader, sending a full tablet report...
I20250902 01:57:16.399135 30622 consensus_queue.cc:260] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.399158 30623 raft_consensus.cc:357] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.400228 30623 raft_consensus.cc:383] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.400616 30624 ts_tablet_manager.cc:1428] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250902 01:57:16.400539 30623 raft_consensus.cc:738] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.401532 30624 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.401422 30623 consensus_queue.cc:260] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.408936 30624 tablet_bootstrap.cc:654] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.421092 30624 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.421659 30624 ts_tablet_manager.cc:1397] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.002s
I20250902 01:57:16.424854 30624 raft_consensus.cc:357] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.425729 30624 raft_consensus.cc:383] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.426117 30624 raft_consensus.cc:738] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.427094 30624 consensus_queue.cc:260] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.431011 30624 ts_tablet_manager.cc:1428] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20250902 01:57:16.432176 30624 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.439849 30624 tablet_bootstrap.cc:654] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.440449 30533 heartbeater.cc:499] Master 127.29.21.254:43255 was elected leader, sending a full tablet report...
I20250902 01:57:16.440632 30607 heartbeater.cc:499] Master 127.29.21.254:43255 was elected leader, sending a full tablet report...
I20250902 01:57:16.441103 30623 ts_tablet_manager.cc:1428] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.044s user 0.032s sys 0.011s
I20250902 01:57:16.442023 30623 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.442718 30622 ts_tablet_manager.cc:1428] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.046s user 0.020s sys 0.023s
I20250902 01:57:16.443626 30622 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.449601 30622 tablet_bootstrap.cc:654] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.451236 30623 tablet_bootstrap.cc:654] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.455593 30624 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.456192 30624 ts_tablet_manager.cc:1397] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.004s
I20250902 01:57:16.459777 30624 raft_consensus.cc:357] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.460712 30624 raft_consensus.cc:383] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.461045 30624 raft_consensus.cc:738] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.461833 30624 consensus_queue.cc:260] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.462685 30623 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.463122 30623 ts_tablet_manager.cc:1397] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.003s
I20250902 01:57:16.464260 30624 ts_tablet_manager.cc:1428] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250902 01:57:16.465075 30624 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.465551 30623 raft_consensus.cc:357] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.466131 30623 raft_consensus.cc:383] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.466369 30623 raft_consensus.cc:738] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.467154 30623 consensus_queue.cc:260] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.469307 30623 ts_tablet_manager.cc:1428] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250902 01:57:16.470134 30623 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.471484 30624 tablet_bootstrap.cc:654] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.477348 30623 tablet_bootstrap.cc:654] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.482983 30624 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.483631 30624 ts_tablet_manager.cc:1397] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20250902 01:57:16.483920 30623 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.484323 30623 ts_tablet_manager.cc:1397] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250902 01:57:16.486156 30624 raft_consensus.cc:357] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.486922 30624 raft_consensus.cc:383] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.486671 30623 raft_consensus.cc:357] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.487349 30624 raft_consensus.cc:738] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.487370 30623 raft_consensus.cc:383] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.487906 30623 raft_consensus.cc:738] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.488386 30624 consensus_queue.cc:260] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.488670 30623 consensus_queue.cc:260] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.490633 30624 ts_tablet_manager.cc:1428] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250902 01:57:16.490633 30623 ts_tablet_manager.cc:1428] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250902 01:57:16.491664 30623 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.493508 30624 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.496815 30622 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.497301 30623 tablet_bootstrap.cc:654] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.498003 30622 ts_tablet_manager.cc:1397] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.055s user 0.033s sys 0.020s
I20250902 01:57:16.499207 30624 tablet_bootstrap.cc:654] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.500536 30622 raft_consensus.cc:357] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.501166 30622 raft_consensus.cc:383] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.501430 30622 raft_consensus.cc:738] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.502136 30622 consensus_queue.cc:260] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.504313 30622 ts_tablet_manager.cc:1428] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:16.505144 30622 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.510805 30622 tablet_bootstrap.cc:654] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.511322 30623 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.511931 30623 ts_tablet_manager.cc:1397] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.002s
I20250902 01:57:16.515324 30623 raft_consensus.cc:357] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.517083 30623 raft_consensus.cc:383] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.517416 30623 raft_consensus.cc:738] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.518345 30624 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.518115 30623 consensus_queue.cc:260] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.518715 30624 ts_tablet_manager.cc:1397] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s
I20250902 01:57:16.520514 30623 ts_tablet_manager.cc:1428] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250902 01:57:16.521411 30623 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.521112 30624 raft_consensus.cc:357] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.521790 30624 raft_consensus.cc:383] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.522063 30624 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.522301 30622 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.522758 30622 ts_tablet_manager.cc:1397] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.002s
I20250902 01:57:16.522838 30624 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.525475 30622 raft_consensus.cc:357] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.526050 30622 raft_consensus.cc:383] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.526306 30622 raft_consensus.cc:738] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.526711 30624 ts_tablet_manager.cc:1428] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20250902 01:57:16.526890 30622 consensus_queue.cc:260] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.527611 30624 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.528026 30623 tablet_bootstrap.cc:654] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.528980 30622 ts_tablet_manager.cc:1428] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250902 01:57:16.533200 30622 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.534494 30624 tablet_bootstrap.cc:654] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.538514 30622 tablet_bootstrap.cc:654] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.539762 30623 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.540349 30623 ts_tablet_manager.cc:1397] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20250902 01:57:16.543581 30623 raft_consensus.cc:357] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.544389 30623 raft_consensus.cc:383] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.544703 30623 raft_consensus.cc:738] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.545404 30623 consensus_queue.cc:260] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.545847 30624 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.546298 30624 ts_tablet_manager.cc:1397] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.011s
I20250902 01:57:16.547788 30623 ts_tablet_manager.cc:1428] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:16.548688 30623 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.548599 30624 raft_consensus.cc:357] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.549238 30624 raft_consensus.cc:383] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.549587 30624 raft_consensus.cc:738] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.550287 30624 consensus_queue.cc:260] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.554571 30624 ts_tablet_manager.cc:1428] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250902 01:57:16.555399 30624 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.556452 30623 tablet_bootstrap.cc:654] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.558028 30622 tablet_bootstrap.cc:492] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.558424 30622 ts_tablet_manager.cc:1397] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.012s
I20250902 01:57:16.560734 30622 raft_consensus.cc:357] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.561372 30622 raft_consensus.cc:383] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.561661 30622 raft_consensus.cc:738] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.561789 30624 tablet_bootstrap.cc:654] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.562407 30622 consensus_queue.cc:260] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.568312 30623 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.568897 30623 ts_tablet_manager.cc:1397] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20250902 01:57:16.569995 30622 ts_tablet_manager.cc:1428] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20250902 01:57:16.571481 30623 raft_consensus.cc:357] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.572163 30623 raft_consensus.cc:383] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.572474 30623 raft_consensus.cc:738] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.572714 30624 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.571856 30622 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.573139 30624 ts_tablet_manager.cc:1397] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250902 01:57:16.573148 30623 consensus_queue.cc:260] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.575629 30624 raft_consensus.cc:357] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.576282 30624 raft_consensus.cc:383] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.576577 30624 raft_consensus.cc:738] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.577240 30623 ts_tablet_manager.cc:1428] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250902 01:57:16.577236 30624 consensus_queue.cc:260] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.578097 30623 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.578743 30622 tablet_bootstrap.cc:654] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.581637 30624 ts_tablet_manager.cc:1428] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250902 01:57:16.582867 30624 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.583801 30623 tablet_bootstrap.cc:654] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.590168 30624 tablet_bootstrap.cc:654] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.595960 30622 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.596386 30622 ts_tablet_manager.cc:1397] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20250902 01:57:16.596608 30624 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.596961 30624 ts_tablet_manager.cc:1397] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250902 01:57:16.598537 30622 raft_consensus.cc:357] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.598753 30624 raft_consensus.cc:357] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.599123 30622 raft_consensus.cc:383] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.599256 30624 raft_consensus.cc:383] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.599416 30622 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.599517 30624 raft_consensus.cc:738] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.600080 30622 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.600216 30624 consensus_queue.cc:260] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.602175 30622 ts_tablet_manager.cc:1428] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:16.603184 30622 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.604259 30624 ts_tablet_manager.cc:1428] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250902 01:57:16.605042 30624 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.608531 30622 tablet_bootstrap.cc:654] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.609843 30624 tablet_bootstrap.cc:654] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.618309 30624 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.618753 30624 ts_tablet_manager.cc:1397] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250902 01:57:16.621517 30624 raft_consensus.cc:357] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.622334 30624 raft_consensus.cc:383] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.622684 30624 raft_consensus.cc:738] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.624979 30628 raft_consensus.cc:491] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.625377 30628 raft_consensus.cc:513] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.627526 30628 leader_election.cc:290] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.628460 30622 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.628886 30622 ts_tablet_manager.cc:1397] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.011s
I20250902 01:57:16.631480 30622 raft_consensus.cc:357] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.632308 30622 raft_consensus.cc:383] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.632671 30622 raft_consensus.cc:738] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.633491 30622 consensus_queue.cc:260] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.636358 30623 tablet_bootstrap.cc:492] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.636685 30623 ts_tablet_manager.cc:1397] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.059s user 0.018s sys 0.038s
I20250902 01:57:16.638413 30622 ts_tablet_manager.cc:1428] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250902 01:57:16.638907 30623 raft_consensus.cc:357] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.639559 30623 raft_consensus.cc:383] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.639693 30622 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.639845 30623 raft_consensus.cc:738] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.640594 30623 consensus_queue.cc:260] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.642833 30623 ts_tablet_manager.cc:1428] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:16.643741 30623 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.647001 30622 tablet_bootstrap.cc:654] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.624560 30624 consensus_queue.cc:260] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.649217 30623 tablet_bootstrap.cc:654] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.650595 30630 raft_consensus.cc:491] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.651260 30628 raft_consensus.cc:491] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.651484 30624 ts_tablet_manager.cc:1428] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250902 01:57:16.651714 30628 raft_consensus.cc:513] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.652325 30624 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: Bootstrap starting.
I20250902 01:57:16.653465 30628 leader_election.cc:290] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.650995 30630 raft_consensus.cc:513] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.657487 30630 leader_election.cc:290] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.673583 30653 raft_consensus.cc:491] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.674775 30653 raft_consensus.cc:513] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.676918 30628 raft_consensus.cc:491] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.677594 30628 raft_consensus.cc:513] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.679848 30628 leader_election.cc:290] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.681977 30624 tablet_bootstrap.cc:654] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.692878 30653 leader_election.cc:290] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5437b7338b794400b09a7d407549227a (127.29.21.195:40753), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.693598 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.694492 30433 raft_consensus.cc:2466] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 0.
I20250902 01:57:16.696269 30471 leader_election.cc:304] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.696494 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.694995 30622 tablet_bootstrap.cc:492] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.697079 30630 raft_consensus.cc:2802] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.698314 30622 ts_tablet_manager.cc:1397] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.013s
W20250902 01:57:16.698761 30473 leader_election.cc:343] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.698484 30630 raft_consensus.cc:491] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.699928 30630 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.702111 30622 raft_consensus.cc:357] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.703125 30622 raft_consensus.cc:383] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.703608 30622 raft_consensus.cc:738] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.704602 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.704666 30630 raft_consensus.cc:513] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.705544 30582 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.706604 30398 leader_election.cc:304] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.706962 30630 leader_election.cc:290] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.704609 30622 consensus_queue.cc:260] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.708122 30581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.708263 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.709093 30580 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.709833 30622 ts_tablet_manager.cc:1428] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250902 01:57:16.711292 30622 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.711922 30653 raft_consensus.cc:2802] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.711026 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.712955 30653 raft_consensus.cc:491] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.713763 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.714715 30653 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.716910 30623 tablet_bootstrap.cc:492] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.717406 30623 ts_tablet_manager.cc:1397] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.074s user 0.024s sys 0.012s
W20250902 01:57:16.715271 30398 leader_election.cc:343] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.718899 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.719349 30398 leader_election.cc:304] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.721064 30628 raft_consensus.cc:2802] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.721385 30628 raft_consensus.cc:491] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.718248 30433 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.721720 30628 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.724534 30653 raft_consensus.cc:513] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.726593 30653 leader_election.cc:290] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.715018 30582 raft_consensus.cc:2466] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.716089 30630 raft_consensus.cc:491] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.728065 30433 raft_consensus.cc:2466] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 1.
I20250902 01:57:16.728377 30630 raft_consensus.cc:513] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.728844 30622 tablet_bootstrap.cc:654] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.728734 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.729202 30471 leader_election.cc:304] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.729614 30398 leader_election.cc:304] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.730110 30630 leader_election.cc:290] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.729218 30582 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.730420 30653 raft_consensus.cc:491] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.730675 30630 raft_consensus.cc:491] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.730821 30653 raft_consensus.cc:513] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.731101 30630 raft_consensus.cc:513] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.732656 30653 leader_election.cc:290] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.733232 30653 raft_consensus.cc:2762] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250902 01:57:16.733449 30630 leader_election.cc:290] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.734266 30630 raft_consensus.cc:2802] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.734752 30630 raft_consensus.cc:695] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:16.735042 30628 raft_consensus.cc:513] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.735776 30630 consensus_queue.cc:237] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.737052 30628 leader_election.cc:290] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Requested vote from peers 5437b7338b794400b09a7d407549227a (127.29.21.195:40753), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.743808 30623 raft_consensus.cc:357] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.744601 30623 raft_consensus.cc:383] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.744900 30623 raft_consensus.cc:738] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.745631 30623 consensus_queue.cc:260] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.749629 30623 ts_tablet_manager.cc:1428] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250902 01:57:16.751204 30582 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.753901 30623 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.755740 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.755808 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.754161 30624 tablet_bootstrap.cc:492] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: No bootstrap required, opened a new log
I20250902 01:57:16.756459 30582 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.756596 30433 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 0.
I20250902 01:57:16.758289 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.758790 30582 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.758877 30659 raft_consensus.cc:491] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.759383 30659 raft_consensus.cc:513] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.761132 30659 leader_election.cc:290] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.762131 30659 raft_consensus.cc:491] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.761602 30471 leader_election.cc:304] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.763644 30624 ts_tablet_manager.cc:1397] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent bootstrapping tablet: real 0.111s user 0.013s sys 0.009s
I20250902 01:57:16.764858 30508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.765012 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.765451 30508 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.765621 30432 raft_consensus.cc:2466] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 0.
I20250902 01:57:16.766141 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:16.766191 30624 raft_consensus.cc:357] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.766786 30504 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.766904 30624 raft_consensus.cc:383] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.758289 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.767277 30506 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.763031 30659 raft_consensus.cc:513] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.767980 30506 raft_consensus.cc:2466] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.769558 30659 leader_election.cc:290] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.767792 30433 raft_consensus.cc:2391] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 673abf5ce13140a7b854b7f50e0f6488 in current term 1: Already voted for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in this term.
I20250902 01:57:16.769349 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.771054 30505 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.771471 30623 tablet_bootstrap.cc:654] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.767189 30471 leader_election.cc:304] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.770455 30629 raft_consensus.cc:491] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.772619 30629 raft_consensus.cc:513] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.773646 30622 tablet_bootstrap.cc:492] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.774284 30622 ts_tablet_manager.cc:1397] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.063s user 0.023s sys 0.016s
I20250902 01:57:16.774471 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.774868 30629 leader_election.cc:290] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.774957 30505 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 0.
I20250902 01:57:16.766465 30507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.777127 30622 raft_consensus.cc:357] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
W20250902 01:57:16.777686 30379 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:16.777897 30622 raft_consensus.cc:383] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.778246 30622 raft_consensus.cc:738] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.779824 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.773707 30504 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.781214 30659 raft_consensus.cc:2802] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.781802 30659 raft_consensus.cc:491] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.784005 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.770246 30398 leader_election.cc:304] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.785091 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:16.767329 30662 raft_consensus.cc:2802] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.785549 30504 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.785687 30660 raft_consensus.cc:2802] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.785923 30662 raft_consensus.cc:491] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.786162 30660 raft_consensus.cc:695] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:16.786375 30662 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.786886 30660 consensus_queue.cc:237] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.782301 30659 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.788225 30328 catalog_manager.cc:5582] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:16.790738 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.785714 30398 leader_election.cc:304] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.794732 30628 raft_consensus.cc:2802] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.795152 30628 raft_consensus.cc:491] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.795521 30628 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.795885 30624 raft_consensus.cc:738] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.796027 30659 raft_consensus.cc:513] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.796538 30624 consensus_queue.cc:260] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.797310 30433 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 0.
I20250902 01:57:16.798291 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.798497 30629 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.798929 30629 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.799556 30659 leader_election.cc:290] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.800386 30396 leader_election.cc:304] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters: 5437b7338b794400b09a7d407549227a
I20250902 01:57:16.800678 30629 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.800861 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:16.798933 30471 leader_election.cc:304] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
W20250902 01:57:16.801390 30473 leader_election.cc:343] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.801628 30580 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 0.
I20250902 01:57:16.802289 30473 leader_election.cc:304] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673abf5ce13140a7b854b7f50e0f6488; no voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a
I20250902 01:57:16.802685 30659 raft_consensus.cc:2802] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.803364 30659 raft_consensus.cc:491] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.803659 30628 raft_consensus.cc:513] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.804872 30653 raft_consensus.cc:2802] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.803887 30659 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.805876 30653 raft_consensus.cc:491] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.805902 30582 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.806383 30653 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.812106 30659 raft_consensus.cc:513] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.812778 30504 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.813248 30630 raft_consensus.cc:2747] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:16.813495 30628 leader_election.cc:290] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.779076 30622 consensus_queue.cc:260] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.815657 30659 leader_election.cc:290] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.820091 30622 ts_tablet_manager.cc:1428] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250902 01:57:16.820938 30622 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.823222 30330 catalog_manager.cc:5582] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:16.825980 30622 tablet_bootstrap.cc:654] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.828140 30624 ts_tablet_manager.cc:1428] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: Time spent starting tablet: real 0.064s user 0.004s sys 0.002s
I20250902 01:57:16.831890 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.832201 30629 raft_consensus.cc:491] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:16.832465 30433 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.832670 30629 raft_consensus.cc:513] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.832659 30653 raft_consensus.cc:513] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.834591 30629 leader_election.cc:290] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.835879 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.836390 30432 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:16.803458 30473 leader_election.cc:343] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.835340 30653 leader_election.cc:290] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.846272 30396 leader_election.cc:304] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.847472 30653 raft_consensus.cc:2802] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.848043 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.848421 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.848598 30580 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.848824 30653 raft_consensus.cc:695] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:16.852536 30507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:16.853073 30507 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.848057 30432 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 1.
I20250902 01:57:16.859215 30662 raft_consensus.cc:513] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.859704 30471 leader_election.cc:304] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.860476 30659 raft_consensus.cc:2802] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.860880 30659 raft_consensus.cc:695] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:16.861146 30662 leader_election.cc:290] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:16.861725 30659 consensus_queue.cc:237] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.865363 30653 consensus_queue.cc:237] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:16.868769 30623 tablet_bootstrap.cc:492] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:16.869153 30623 ts_tablet_manager.cc:1397] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.115s user 0.015s sys 0.019s
I20250902 01:57:16.872355 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.873579 30623 raft_consensus.cc:357] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.873965 30623 raft_consensus.cc:383] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.874150 30623 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.874766 30623 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.875101 30581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
W20250902 01:57:16.873095 30473 leader_election.cc:343] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.875905 30581 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.875121 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.872905 30432 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.872653 30433 raft_consensus.cc:2466] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 1.
I20250902 01:57:16.872985 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:16.880982 30623 ts_tablet_manager.cc:1428] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250902 01:57:16.877818 30582 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.882591 30623 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: Bootstrap starting.
I20250902 01:57:16.881723 30430 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.883500 30430 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 0.
I20250902 01:57:16.883746 30471 leader_election.cc:304] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters: 5437b7338b794400b09a7d407549227a
I20250902 01:57:16.884824 30431 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.885035 30630 raft_consensus.cc:2802] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.885418 30630 raft_consensus.cc:695] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:16.885390 30431 raft_consensus.cc:2466] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 0.
I20250902 01:57:16.879856 30504 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.881903 30507 raft_consensus.cc:2466] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.888706 30545 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.889398 30629 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.889760 30629 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.889519 30630 consensus_queue.cc:237] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.890067 30629 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.891832 30396 leader_election.cc:304] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.893573 30668 raft_consensus.cc:2802] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.894125 30668 raft_consensus.cc:695] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:16.885965 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.894948 30668 consensus_queue.cc:237] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.899607 30579 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:16.899873 30507 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.900962 30507 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 0.
W20250902 01:57:16.901563 30473 leader_election.cc:343] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5437b7338b794400b09a7d407549227a (127.29.21.195:40753): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.902127 30545 leader_election.cc:304] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:16.903625 30429 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:16.910257 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:16.910919 30505 raft_consensus.cc:2391] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5437b7338b794400b09a7d407549227a in current term 1: Already voted for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in this term.
I20250902 01:57:16.926360 30622 tablet_bootstrap.cc:492] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:16.926779 30622 ts_tablet_manager.cc:1397] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.106s user 0.010s sys 0.031s
I20250902 01:57:16.927194 30580 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 1.
I20250902 01:57:16.929659 30622 raft_consensus.cc:357] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.930395 30622 raft_consensus.cc:383] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:16.930657 30622 raft_consensus.cc:738] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:16.931392 30622 consensus_queue.cc:260] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.942198 30623 tablet_bootstrap.cc:654] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.949524 30582 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.952965 30432 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 1.
I20250902 01:57:16.954187 30471 leader_election.cc:304] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters: 5437b7338b794400b09a7d407549227a
I20250902 01:57:16.955525 30659 raft_consensus.cc:2802] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.956064 30659 raft_consensus.cc:695] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:16.957592 30659 consensus_queue.cc:237] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.941749 30329 catalog_manager.cc:5582] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:16.969220 30504 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.970564 30396 leader_election.cc:304] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:16.974152 30629 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.962749 30581 raft_consensus.cc:2466] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 1.
I20250902 01:57:16.977634 30622 ts_tablet_manager.cc:1428] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.051s user 0.013s sys 0.012s
I20250902 01:57:16.979034 30677 raft_consensus.cc:2802] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:16.979424 30677 raft_consensus.cc:491] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:16.979792 30677 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.980437 30622 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: Bootstrap starting.
I20250902 01:57:16.986001 30622 tablet_bootstrap.cc:654] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:16.986159 30677 raft_consensus.cc:513] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:16.989743 30677 leader_election.cc:290] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:16.988008 30653 raft_consensus.cc:2802] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:16.992137 30629 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
W20250902 01:57:16.988260 30545 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:33507, user_credentials={real_user=slave}} blocked reactor thread for 91835.4us
I20250902 01:57:16.998373 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.997895 30653 raft_consensus.cc:695] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
W20250902 01:57:16.998935 30545 leader_election.cc:343] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:16.998979 30432 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.999084 30429 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:16.999665 30429 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:16.999528 30653 consensus_queue.cc:237] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.000123 30545 leader_election.cc:304] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5437b7338b794400b09a7d407549227a; no voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:17.001785 30629 raft_consensus.cc:2747] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:17.002599 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:17.003142 30505 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:17.007191 30432 raft_consensus.cc:2466] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 1.
I20250902 01:57:17.007802 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:16.995043 30330 catalog_manager.cc:5582] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.011878 30623 tablet_bootstrap.cc:492] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: No bootstrap required, opened a new log
I20250902 01:57:17.012183 30623 ts_tablet_manager.cc:1397] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: Time spent bootstrapping tablet: real 0.130s user 0.012s sys 0.006s
I20250902 01:57:17.007496 30429 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 1.
I20250902 01:57:17.014534 30623 raft_consensus.cc:357] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.014961 30623 raft_consensus.cc:383] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:17.015142 30623 raft_consensus.cc:738] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Initialized, Role: FOLLOWER
I20250902 01:57:17.015650 30623 consensus_queue.cc:260] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
W20250902 01:57:17.017288 30545 leader_election.cc:343] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:17.018126 30545 leader_election.cc:304] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters: 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:17.018795 30629 raft_consensus.cc:2802] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:17.019213 30629 raft_consensus.cc:695] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:17.019958 30629 consensus_queue.cc:237] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.023897 30329 catalog_manager.cc:5582] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.026226 30329 catalog_manager.cc:5582] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.028772 30623 ts_tablet_manager.cc:1428] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250902 01:57:17.032994 30505 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 1.
I20250902 01:57:17.029480 30545 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:17.035733 30675 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:17.036490 30675 raft_consensus.cc:695] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:17.037992 30675 consensus_queue.cc:237] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.043733 30330 catalog_manager.cc:5582] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.045928 30330 catalog_manager.cc:5582] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.049372 30622 tablet_bootstrap.cc:492] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: No bootstrap required, opened a new log
I20250902 01:57:17.049069 30328 catalog_manager.cc:5582] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 1 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:17.049703 30622 ts_tablet_manager.cc:1397] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.012s
I20250902 01:57:17.051421 30622 raft_consensus.cc:357] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.051887 30622 raft_consensus.cc:383] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:17.052075 30622 raft_consensus.cc:738] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Initialized, Role: FOLLOWER
I20250902 01:57:17.052582 30622 consensus_queue.cc:260] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.053973 30622 ts_tablet_manager.cc:1428] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250902 01:57:17.065032 30328 catalog_manager.cc:5582] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 1 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:17.159634 30581 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "5437b7338b794400b09a7d407549227a"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:45868
I20250902 01:57:17.160072 30581 raft_consensus.cc:604] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:17.169519 30505 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "30272acb192f4b15864059546a8019a3"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50724
I20250902 01:57:17.169875 30505 raft_consensus.cc:604] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:17.180375 30581 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d"
dest_uuid: "5437b7338b794400b09a7d407549227a"
mode: GRACEFUL
new_leader_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:45868
I20250902 01:57:17.180775 30581 raft_consensus.cc:604] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Received request to transfer leadership to TS 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:17.190550 30429 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:50270
I20250902 01:57:17.190994 30429 raft_consensus.cc:604] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Received request to transfer leadership to TS 5437b7338b794400b09a7d407549227a
I20250902 01:57:17.194525 30629 raft_consensus.cc:491] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:17.194865 30629 raft_consensus.cc:513] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.196221 30629 leader_election.cc:290] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:17.196925 30429 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:17.197068 30505 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673abf5ce13140a7b854b7f50e0f6488" is_pre_election: true
I20250902 01:57:17.197505 30429 raft_consensus.cc:2391] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5437b7338b794400b09a7d407549227a in current term 1: Already voted for candidate 673abf5ce13140a7b854b7f50e0f6488 in this term.
I20250902 01:57:17.198343 30545 leader_election.cc:304] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5437b7338b794400b09a7d407549227a; no voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:17.198868 30629 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:17.203037 30629 raft_consensus.cc:2747] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:17.225979 30659 consensus_queue.cc:1035] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.229954 30581 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:17.240044 30662 consensus_queue.cc:1035] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.291462 30662 consensus_queue.cc:1035] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.301013 30630 consensus_queue.cc:1035] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.309742 30653 consensus_queue.cc:1035] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.311769 30653 consensus_queue.cc:1035] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.322865 30628 consensus_queue.cc:1035] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.327667 30653 consensus_queue.cc:1035] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.344168 30630 consensus_queue.cc:1035] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.349681 30659 raft_consensus.cc:991] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:17.350070 30659 raft_consensus.cc:1079] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:17.351473 30432 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "30272acb192f4b15864059546a8019a3"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50254
I20250902 01:57:17.352628 30432 raft_consensus.cc:491] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:17.352962 30432 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.355649 30630 consensus_queue.cc:1035] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.358706 30432 raft_consensus.cc:513] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.363595 30432 leader_election.cc:290] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:17.365199 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:17.365689 30504 raft_consensus.cc:3053] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:17.365990 30504 raft_consensus.cc:738] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:17.366712 30504 consensus_queue.cc:260] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.368005 30504 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.364605 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30272acb192f4b15864059546a8019a3" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:17.369104 30582 raft_consensus.cc:3058] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.374663 30504 raft_consensus.cc:2466] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:17.375433 30582 raft_consensus.cc:2466] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:17.375638 30396 leader_election.cc:304] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:17.376271 30628 raft_consensus.cc:2802] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:17.376681 30628 raft_consensus.cc:695] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:17.377271 30628 consensus_queue.cc:237] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.382489 30329 catalog_manager.cc:5582] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 1 to 2, leader changed from 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.399201 30582 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:17.403995 30630 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.412654 30662 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.451272 30662 raft_consensus.cc:491] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:17.451766 30662 raft_consensus.cc:513] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.453279 30662 leader_election.cc:290] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:17.453281 30628 consensus_queue.cc:1035] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:17.453889 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" is_pre_election: true
I20250902 01:57:17.454254 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5437b7338b794400b09a7d407549227a" is_pre_election: true
I20250902 01:57:17.454568 30432 raft_consensus.cc:2391] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 673abf5ce13140a7b854b7f50e0f6488 in current term 1: Already voted for candidate 5437b7338b794400b09a7d407549227a in this term.
I20250902 01:57:17.455464 30473 leader_election.cc:304] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 673abf5ce13140a7b854b7f50e0f6488; no voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a
I20250902 01:57:17.456127 30662 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:17.462317 30662 raft_consensus.cc:2747] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:17.464097 30668 consensus_queue.cc:1035] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.470849 30668 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.475404 30668 raft_consensus.cc:991] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:17.475826 30668 raft_consensus.cc:1079] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Signalling peer 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:17.477347 30581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf"
dest_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:45852
I20250902 01:57:17.477794 30581 raft_consensus.cc:491] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:17.478113 30581 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.480559 30628 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.489308 30581 raft_consensus.cc:513] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.490729 30581 leader_election.cc:290] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:17.491458 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:17.491962 30432 raft_consensus.cc:3053] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:17.492161 30432 raft_consensus.cc:738] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:17.492771 30432 consensus_queue.cc:260] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.493597 30432 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.496697 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d404cc6e160a41458b944ce1811659cf" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:17.497161 30504 raft_consensus.cc:3058] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.497673 30432 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:17.498420 30545 leader_election.cc:304] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:17.498946 30629 raft_consensus.cc:2802] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:17.499274 30629 raft_consensus.cc:695] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:17.499884 30629 consensus_queue.cc:237] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.502221 30504 raft_consensus.cc:2466] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:17.505080 30329 catalog_manager.cc:5582] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 1 to 2, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:17.549966 30629 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.555517 30675 raft_consensus.cc:991] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:17.556211 30629 raft_consensus.cc:1079] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:17.557674 30429 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50268
I20250902 01:57:17.559049 30429 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:17.559427 30429 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.560490 30675 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.565263 30429 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.567495 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:17.567952 30429 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:17.568013 30504 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.568845 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:17.569383 30582 raft_consensus.cc:3053] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:17.569687 30582 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:17.570417 30582 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.571635 30582 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.573282 30504 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:17.574376 30396 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:17.575167 30668 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:17.575630 30668 raft_consensus.cc:695] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:17.576323 30668 consensus_queue.cc:237] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.577433 30582 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:17.581578 30330 catalog_manager.cc:5582] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 1 to 2, leader changed from 5437b7338b794400b09a7d407549227a (127.29.21.195) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.622694 30706 consensus_queue.cc:1035] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.628275 30675 raft_consensus.cc:991] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: : Instructing follower 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:17.628947 30629 raft_consensus.cc:1079] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Signalling peer 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:17.630285 30505 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:50714
I20250902 01:57:17.632824 30505 raft_consensus.cc:491] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:17.633210 30505 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.633332 30629 consensus_queue.cc:1035] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:17.639273 30505 raft_consensus.cc:513] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.642675 30505 leader_election.cc:290] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 2 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:17.643507 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:17.643759 30582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb10f324c60e4436851b4abbcf2a503d" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:17.644045 30432 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.644219 30582 raft_consensus.cc:3053] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:17.644522 30582 raft_consensus.cc:738] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:17.645164 30582 consensus_queue.cc:260] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.645934 30582 raft_consensus.cc:3058] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:17.648555 30432 raft_consensus.cc:2466] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 2.
I20250902 01:57:17.649369 30471 leader_election.cc:304] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:17.650071 30662 raft_consensus.cc:2802] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:17.650120 30582 raft_consensus.cc:2466] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 2.
I20250902 01:57:17.650543 30662 raft_consensus.cc:695] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 2 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:17.651269 30662 consensus_queue.cc:237] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:17.656441 30329 catalog_manager.cc:5582] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 1 to 2, leader changed from 5437b7338b794400b09a7d407549227a (127.29.21.195) to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:17.897704 30582 raft_consensus.cc:1273] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:17.898852 30668 consensus_queue.cc:1035] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:17.906327 30505 raft_consensus.cc:1273] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:17.907874 30660 consensus_queue.cc:1035] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:18.004164 30505 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.005126 30660 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:18.010541 30432 raft_consensus.cc:1273] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.011601 30582 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.012814 30706 consensus_queue.cc:1035] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:18.012988 30660 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:57:18.021413 30505 raft_consensus.cc:1273] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.022581 30629 consensus_queue.cc:1035] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:18.174515 30582 raft_consensus.cc:1273] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.175617 30659 consensus_queue.cc:1035] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:18.178642 30432 raft_consensus.cc:1273] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:18.179908 30659 consensus_queue.cc:1035] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:20.216497 29783 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250902 01:57:20.217216 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:20.237135 30432 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:20.237555 30432 raft_consensus.cc:604] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Received request to transfer leadership to TS 5437b7338b794400b09a7d407549227a
I20250902 01:57:20.239301 30432 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:20.239719 30432 raft_consensus.cc:604] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Received request to transfer leadership to TS 5437b7338b794400b09a7d407549227a
I20250902 01:57:20.240494 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250902 01:57:20.240852 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:20.314060 30710 raft_consensus.cc:991] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:20.314446 30660 raft_consensus.cc:1079] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Signalling peer 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:20.315600 30582 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:45852
I20250902 01:57:20.315954 30582 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:20.316169 30582 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:20.320477 30582 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:20.321698 30582 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 3 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:20.322430 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:20.322696 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:20.323007 30432 raft_consensus.cc:3053] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:20.323197 30504 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:20.323357 30432 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:20.324159 30432 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:20.325259 30432 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:20.329475 30504 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 3.
I20250902 01:57:20.330574 30545 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:20.331779 30432 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 3.
I20250902 01:57:20.331868 30713 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:20.333068 30713 raft_consensus.cc:695] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:20.333662 30713 consensus_queue.cc:237] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:20.339151 30330 catalog_manager.cc:5582] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 2 to 3, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 3 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:20.614023 30710 raft_consensus.cc:991] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:20.614405 30660 raft_consensus.cc:1079] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Signalling peer 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:20.615566 30582 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04"
dest_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:45852
I20250902 01:57:20.615962 30582 raft_consensus.cc:491] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:20.616175 30582 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:20.619783 30582 raft_consensus.cc:513] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:20.620942 30582 leader_election.cc:290] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:20.621668 30432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:20.621852 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:20.622143 30432 raft_consensus.cc:3053] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:20.622339 30504 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:20.622427 30432 raft_consensus.cc:738] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:20.623054 30432 consensus_queue.cc:260] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:20.623903 30432 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:20.626708 30504 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:20.627503 30545 leader_election.cc:304] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:20.628077 30713 raft_consensus.cc:2802] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:20.628078 30432 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:20.628477 30713 raft_consensus.cc:695] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:20.629175 30713 consensus_queue.cc:237] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:20.634622 30330 catalog_manager.cc:5582] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 1 to 2, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:20.780050 30432 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:20.781047 30713 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:20.788913 30504 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:20.790093 30713 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:21.021082 30504 raft_consensus.cc:1273] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:21.022109 30713 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:21.030252 30432 raft_consensus.cc:1273] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:21.031827 30714 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:21.241559 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:21.247351 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:21.247844 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:22.248629 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:22.253907 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:22.254374 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:23.255282 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:23.261528 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:23.261873 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:24.262705 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:24.269507 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:24.269892 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:25.270938 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:25.276693 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:25.276995 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:26.277860 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:26.282869 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:26.283169 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:27.283941 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:27.288539 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:27.288846 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:28.389336 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:28.527424 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:28.528043 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250902 01:57:29.519192 30454 debug-util.cc:398] Leaking SignalData structure 0x7b080014cb60 after lost signal to thread 29786
W20250902 01:57:29.520107 30454 debug-util.cc:398] Leaking SignalData structure 0x7b080019a3e0 after lost signal to thread 30365
W20250902 01:57:29.520983 30454 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6340 after lost signal to thread 30457
W20250902 01:57:29.521862 30454 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8720 after lost signal to thread 30532
W20250902 01:57:29.522724 30454 debug-util.cc:398] Leaking SignalData structure 0x7b080010a140 after lost signal to thread 30606
I20250902 01:57:29.529059 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:29.537551 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:29.538012 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:30.568043 29783 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250902 01:57:30.571920 30580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04"
dest_uuid: "5437b7338b794400b09a7d407549227a"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:45868
I20250902 01:57:30.572645 30580 raft_consensus.cc:604] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:30.580955 30580 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "5437b7338b794400b09a7d407549227a"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:45868
I20250902 01:57:30.581379 30580 raft_consensus.cc:604] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:30.587523 30504 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50724
I20250902 01:57:30.587913 30504 raft_consensus.cc:604] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:30.591775 30504 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50724
I20250902 01:57:30.592228 30504 raft_consensus.cc:604] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:30.596199 30504 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
mode: GRACEFUL
new_leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50724
I20250902 01:57:30.596650 30504 raft_consensus.cc:604] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Received request to transfer leadership to TS 36d29e8094f34cacad7a80e1f2ddbed3
I20250902 01:57:30.715378 30713 raft_consensus.cc:991] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.715952 30738 raft_consensus.cc:1079] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.717414 30433 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50268
I20250902 01:57:30.717811 30433 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:30.718106 30433 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:30.723312 30433 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.725077 30433 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 4 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:30.725778 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:30.726100 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:30.726522 30504 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:30.726694 30580 raft_consensus.cc:3053] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:57:30.727026 30580 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:30.727718 30580 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.728575 30580 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:57:30.733153 30580 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 4.
I20250902 01:57:30.733209 30504 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 4.
I20250902 01:57:30.734231 30398 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:30.735033 30660 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:57:30.735913 30660 raft_consensus.cc:695] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:30.736807 30660 consensus_queue.cc:237] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.747308 30330 catalog_manager.cc:5582] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 3 to 4, leader changed from 5437b7338b794400b09a7d407549227a (127.29.21.195) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:30.780741 30745 raft_consensus.cc:991] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.781155 30745 raft_consensus.cc:1079] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.782723 30433 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50254
I20250902 01:57:30.783260 30433 raft_consensus.cc:491] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:30.783624 30433 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.790930 30433 raft_consensus.cc:513] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.793133 30433 leader_election.cc:290] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:30.793247 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:30.793599 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43d6dabf0ff34265a47b8b39c2e84266" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:30.793807 30504 raft_consensus.cc:3053] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:30.794054 30504 raft_consensus.cc:738] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:30.794108 30580 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.794725 30504 consensus_queue.cc:260] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.795765 30504 raft_consensus.cc:3058] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.799798 30580 raft_consensus.cc:2466] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:30.800657 30398 leader_election.cc:304] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:30.801376 30660 raft_consensus.cc:2802] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:30.801460 30504 raft_consensus.cc:2466] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:30.801882 30660 raft_consensus.cc:695] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:30.802726 30660 consensus_queue.cc:237] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.809435 30330 catalog_manager.cc:5582] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 1 to 2, leader changed from 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:30.944439 30733 raft_consensus.cc:991] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.944949 30745 raft_consensus.cc:1079] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:30.946187 30433 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50254
I20250902 01:57:30.946592 30433 raft_consensus.cc:491] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:30.946844 30433 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.950977 30433 raft_consensus.cc:513] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.952462 30433 leader_election.cc:290] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:30.953114 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:30.953590 30504 raft_consensus.cc:3053] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:30.953547 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:30.953967 30504 raft_consensus.cc:738] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:30.954137 30580 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.954684 30504 consensus_queue.cc:260] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.955835 30504 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:30.959923 30580 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:30.960785 30398 leader_election.cc:304] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:30.961474 30504 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:30.961549 30660 raft_consensus.cc:2802] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:30.963416 30660 raft_consensus.cc:695] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:30.964267 30660 consensus_queue.cc:237] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:30.971432 30330 catalog_manager.cc:5582] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 1 to 2, leader changed from 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:31.002975 30733 raft_consensus.cc:991] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:31.003389 30745 raft_consensus.cc:1079] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:31.004508 30433 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50254
I20250902 01:57:31.005146 30433 raft_consensus.cc:491] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:31.005554 30433 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:31.013000 30433 raft_consensus.cc:513] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:31.014742 30433 leader_election.cc:290] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Requested vote from peers 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:31.015229 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:31.015703 30504 raft_consensus.cc:3053] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:31.015760 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12b7b1fd6ef4af2a1a7e3917a553838" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:31.016114 30504 raft_consensus.cc:738] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:31.016302 30580 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:31.016819 30504 consensus_queue.cc:260] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:31.017853 30504 raft_consensus.cc:3058] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:31.022209 30580 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:31.023105 30398 leader_election.cc:304] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 5437b7338b794400b09a7d407549227a; no voters:
I20250902 01:57:31.023667 30504 raft_consensus.cc:2466] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 2.
I20250902 01:57:31.023882 30660 raft_consensus.cc:2802] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:31.024470 30660 raft_consensus.cc:695] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:31.025463 30660 consensus_queue.cc:237] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:31.032056 30330 catalog_manager.cc:5582] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 1 to 2, leader changed from 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:31.091795 30713 raft_consensus.cc:991] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: : Instructing follower 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:31.092190 30738 raft_consensus.cc:1079] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Signalling peer 36d29e8094f34cacad7a80e1f2ddbed3 to start an election
I20250902 01:57:31.093533 30433 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
from {username='slave'} at 127.0.0.1:50268
I20250902 01:57:31.094029 30433 raft_consensus.cc:491] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:31.094333 30433 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:31.100980 30433 raft_consensus.cc:513] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:31.102932 30433 leader_election.cc:290] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 3 election: Requested vote from peers 5437b7338b794400b09a7d407549227a (127.29.21.195:40753), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:31.103709 30504 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:31.103777 30580 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef2dd292e59f4e28b6349b96fdba1b04" candidate_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:31.104172 30504 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:31.104336 30580 raft_consensus.cc:3053] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:31.104703 30580 raft_consensus.cc:738] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:31.105557 30580 consensus_queue.cc:260] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:31.106789 30580 raft_consensus.cc:3058] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:31.108090 30504 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 3.
I20250902 01:57:31.108958 30396 leader_election.cc:304] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36d29e8094f34cacad7a80e1f2ddbed3, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:31.109565 30660 raft_consensus.cc:2802] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:31.109886 30660 raft_consensus.cc:695] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 LEADER]: Becoming Leader. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:31.110561 30660 consensus_queue.cc:237] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } }
I20250902 01:57:31.113920 30580 raft_consensus.cc:2466] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36d29e8094f34cacad7a80e1f2ddbed3 in term 3.
I20250902 01:57:31.118955 30330 catalog_manager.cc:5582] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 reported cstate change: term changed from 2 to 3, leader changed from 5437b7338b794400b09a7d407549227a (127.29.21.195) to 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:31.178790 30504 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 4 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:31.179793 30660 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:31.193320 30580 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 4 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:57:31.194898 30747 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:57:31.306053 30504 raft_consensus.cc:1273] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.307343 30747 consensus_queue.cc:1035] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.315527 30580 raft_consensus.cc:1273] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.318235 30660 consensus_queue.cc:1035] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.435063 30580 raft_consensus.cc:1273] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.436142 30660 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.444461 30504 raft_consensus.cc:1273] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.446120 30739 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.517678 30504 raft_consensus.cc:1273] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.518895 30747 consensus_queue.cc:1035] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.528102 30580 raft_consensus.cc:1273] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:31.529209 30660 consensus_queue.cc:1035] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:31.567061 30580 raft_consensus.cc:1273] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:31.568009 30761 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:31.575981 30504 raft_consensus.cc:1273] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Refusing update from remote peer 36d29e8094f34cacad7a80e1f2ddbed3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:31.577170 30747 consensus_queue.cc:1035] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:33.624686 29783 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250902 01:57:33.625221 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:33.631368 30433 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2746e7eebde8423f81678907aba6af92"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:33.631793 30433 raft_consensus.cc:604] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Received request to transfer leadership to TS 5437b7338b794400b09a7d407549227a
I20250902 01:57:33.633666 30433 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:33.633991 30433 raft_consensus.cc:604] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Received request to transfer leadership to TS 5437b7338b794400b09a7d407549227a
I20250902 01:57:33.635470 30433 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:33.635798 30433 raft_consensus.cc:604] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 LEADER]: Received request to transfer leadership to TS 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:33.637292 30433 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90"
dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
mode: GRACEFUL
new_leader_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:50280
I20250902 01:57:33.637624 30433 raft_consensus.cc:604] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Received request to transfer leadership to TS 673abf5ce13140a7b854b7f50e0f6488
I20250902 01:57:33.638342 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250902 01:57:33.638655 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:33.662501 30765 raft_consensus.cc:991] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:33.662882 30765 raft_consensus.cc:1079] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 LEADER]: Signalling peer 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:33.663888 30508 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:50706
I20250902 01:57:33.664299 30508 raft_consensus.cc:491] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:33.664546 30508 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:33.668867 30508 raft_consensus.cc:513] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.670504 30508 leader_election.cc:290] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 5 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:33.671234 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:33.671512 30581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "777e53e58e4242408e7377ffe8ecb5a8" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:33.671787 30433 raft_consensus.cc:3053] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:57:33.672036 30581 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:33.672072 30433 raft_consensus.cc:738] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:33.672775 30433 consensus_queue.cc:260] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.673548 30433 raft_consensus.cc:3058] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:57:33.676839 30581 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 5.
I20250902 01:57:33.677853 30473 leader_election.cc:304] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:33.678323 30433 raft_consensus.cc:2466] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 5.
I20250902 01:57:33.678406 30745 raft_consensus.cc:2802] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:57:33.679428 30745 raft_consensus.cc:695] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 5 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:33.680169 30745 consensus_queue.cc:237] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.685739 30328 catalog_manager.cc:5582] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 4 to 5, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 5 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:33.767205 30766 raft_consensus.cc:991] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:33.767640 30765 raft_consensus.cc:1079] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Signalling peer 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:33.768766 30581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e"
dest_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:45852
I20250902 01:57:33.769151 30581 raft_consensus.cc:491] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:33.769418 30581 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.773466 30581 raft_consensus.cc:513] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.775074 30581 leader_election.cc:290] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:33.775774 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:33.775952 30508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d12e1a2135c4ebeaf1adf6edb9aa07e" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:33.776230 30433 raft_consensus.cc:3053] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:33.776422 30508 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.776508 30433 raft_consensus.cc:738] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:33.777103 30433 consensus_queue.cc:260] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.777859 30433 raft_consensus.cc:3058] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.780375 30508 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:33.781260 30545 leader_election.cc:304] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:33.781822 30713 raft_consensus.cc:2802] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:33.781890 30433 raft_consensus.cc:2466] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:33.783093 30713 raft_consensus.cc:695] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:33.783759 30713 consensus_queue.cc:237] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.789532 30328 catalog_manager.cc:5582] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 1 to 2, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:33.793768 30765 raft_consensus.cc:991] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:33.794173 30765 raft_consensus.cc:1079] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Signalling peer 5437b7338b794400b09a7d407549227a to start an election
I20250902 01:57:33.795320 30581 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2746e7eebde8423f81678907aba6af92"
dest_uuid: "5437b7338b794400b09a7d407549227a"
from {username='slave'} at 127.0.0.1:45852
I20250902 01:57:33.795720 30581 raft_consensus.cc:491] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:33.796036 30581 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.799836 30581 raft_consensus.cc:513] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.801000 30581 leader_election.cc:290] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194:40563)
I20250902 01:57:33.801605 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:33.801841 30508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2746e7eebde8423f81678907aba6af92" candidate_uuid: "5437b7338b794400b09a7d407549227a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
I20250902 01:57:33.802153 30433 raft_consensus.cc:3053] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:33.802345 30508 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.802501 30433 raft_consensus.cc:738] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:33.803148 30433 consensus_queue.cc:260] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.804301 30433 raft_consensus.cc:3058] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:33.808214 30508 raft_consensus.cc:2466] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:33.808434 30766 raft_consensus.cc:991] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: : Instructing follower 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:33.808864 30766 raft_consensus.cc:1079] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Signalling peer 673abf5ce13140a7b854b7f50e0f6488 to start an election
I20250902 01:57:33.809120 30545 leader_election.cc:304] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:33.809767 30713 raft_consensus.cc:2802] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:33.810026 30508 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90"
dest_uuid: "673abf5ce13140a7b854b7f50e0f6488"
from {username='slave'} at 127.0.0.1:50706
I20250902 01:57:33.810142 30713 raft_consensus.cc:695] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Becoming Leader. State: Replica: 5437b7338b794400b09a7d407549227a, State: Running, Role: LEADER
I20250902 01:57:33.810529 30508 raft_consensus.cc:491] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:57:33.810597 30433 raft_consensus.cc:2466] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5437b7338b794400b09a7d407549227a in term 2.
I20250902 01:57:33.810863 30508 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:33.810891 30713 consensus_queue.cc:237] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.815141 30508 raft_consensus.cc:513] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.816599 30508 leader_election.cc:290] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 3 election: Requested vote from peers 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507), 5437b7338b794400b09a7d407549227a (127.29.21.195:40753)
I20250902 01:57:33.817036 30433 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36d29e8094f34cacad7a80e1f2ddbed3"
I20250902 01:57:33.817474 30433 raft_consensus.cc:3053] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:33.817433 30581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d5844121c7e4b4c968a544c1b568e90" candidate_uuid: "673abf5ce13140a7b854b7f50e0f6488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5437b7338b794400b09a7d407549227a"
I20250902 01:57:33.817814 30433 raft_consensus.cc:738] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36d29e8094f34cacad7a80e1f2ddbed3, State: Running, Role: LEADER
I20250902 01:57:33.818048 30581 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:33.818401 30433 consensus_queue.cc:260] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.818326 30328 catalog_manager.cc:5582] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a reported cstate change: term changed from 1 to 2, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 5437b7338b794400b09a7d407549227a (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5437b7338b794400b09a7d407549227a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:33.819096 30433 raft_consensus.cc:3058] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:33.822654 30581 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 3.
I20250902 01:57:33.823735 30473 leader_election.cc:304] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5437b7338b794400b09a7d407549227a, 673abf5ce13140a7b854b7f50e0f6488; no voters:
I20250902 01:57:33.824476 30745 raft_consensus.cc:2802] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:33.824384 30433 raft_consensus.cc:2466] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 673abf5ce13140a7b854b7f50e0f6488 in term 3.
I20250902 01:57:33.824954 30745 raft_consensus.cc:695] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 LEADER]: Becoming Leader. State: Replica: 673abf5ce13140a7b854b7f50e0f6488, State: Running, Role: LEADER
I20250902 01:57:33.825690 30745 consensus_queue.cc:237] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } }
I20250902 01:57:33.831125 30328 catalog_manager.cc:5582] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 reported cstate change: term changed from 2 to 3, leader changed from 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193) to 673abf5ce13140a7b854b7f50e0f6488 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "673abf5ce13140a7b854b7f50e0f6488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:34.118481 30433 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 5 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250902 01:57:34.119582 30745 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:57:34.128005 30581 raft_consensus.cc:1273] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 5 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250902 01:57:34.129357 30745 consensus_queue.cc:1035] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:57:34.165560 30508 raft_consensus.cc:1273] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:34.166513 30713 consensus_queue.cc:1035] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:34.174345 30433 raft_consensus.cc:1273] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:34.176205 30713 consensus_queue.cc:1035] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:34.213868 30433 raft_consensus.cc:1273] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:34.214854 30779 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:34.222136 30580 raft_consensus.cc:1273] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Refusing update from remote peer 673abf5ce13140a7b854b7f50e0f6488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:57:34.223805 30767 consensus_queue.cc:1035] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5437b7338b794400b09a7d407549227a" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:57:34.256409 30508 raft_consensus.cc:1273] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:34.257542 30713 consensus_queue.cc:1035] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "673abf5ce13140a7b854b7f50e0f6488" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:34.265661 30433 raft_consensus.cc:1273] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Refusing update from remote peer 5437b7338b794400b09a7d407549227a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:34.266798 30784 consensus_queue.cc:1035] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d29e8094f34cacad7a80e1f2ddbed3" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:34.639429 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:34.644160 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:34.644490 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:35.645511 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:35.651355 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:35.651738 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:36.652565 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:36.657555 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:36.657899 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:37.658604 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:37.663820 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:37.664166 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:38.665010 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:38.670946 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:38.671337 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:39.672158 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:39.677090 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:39.677448 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:40.678215 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:40.683050 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:40.683429 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:41.684367 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:41.689237 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:41.689615 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:42.690564 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:57:42.696316 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250902 01:57:42.696739 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:57:43.726055 29783 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250902 01:57:43.726575 29783 tablet_server.cc:178] TabletServer@127.29.21.193:0 shutting down...
W20250902 01:57:43.744501 30545 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250902 01:57:43.751521 30545 consensus_peers.cc:489] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a -> Peer 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507): Couldn't send request to peer 36d29e8094f34cacad7a80e1f2ddbed3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:43.756108 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:43.756856 29783 tablet_replica.cc:331] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.757957 29783 raft_consensus.cc:2241] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.759105 29783 raft_consensus.cc:2270] T ef2dd292e59f4e28b6349b96fdba1b04 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250902 01:57:43.759268 30471 consensus_peers.cc:489] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 -> Peer 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507): Couldn't send request to peer 36d29e8094f34cacad7a80e1f2ddbed3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:43.761225 29783 tablet_replica.cc:331] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.761731 29783 raft_consensus.cc:2241] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.762122 29783 raft_consensus.cc:2270] T 777e53e58e4242408e7377ffe8ecb5a8 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.763863 29783 tablet_replica.cc:331] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.764343 29783 raft_consensus.cc:2241] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.765007 29783 raft_consensus.cc:2270] T c12b7b1fd6ef4af2a1a7e3917a553838 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.766599 29783 tablet_replica.cc:331] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.767066 29783 raft_consensus.cc:2241] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.767482 29783 raft_consensus.cc:2270] T eb10f324c60e4436851b4abbcf2a503d P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.769505 29783 tablet_replica.cc:331] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.769984 29783 raft_consensus.cc:2241] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.770392 29783 raft_consensus.cc:2270] T 2746e7eebde8423f81678907aba6af92 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.772267 29783 tablet_replica.cc:331] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.772702 29783 raft_consensus.cc:2241] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.773417 29783 raft_consensus.cc:2270] T 30272acb192f4b15864059546a8019a3 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.775022 29783 tablet_replica.cc:331] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.775480 29783 raft_consensus.cc:2241] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.776176 29783 raft_consensus.cc:2270] T 43d6dabf0ff34265a47b8b39c2e84266 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.777609 29783 tablet_replica.cc:331] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.778036 29783 raft_consensus.cc:2241] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.778409 29783 raft_consensus.cc:2270] T d404cc6e160a41458b944ce1811659cf P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.779886 29783 tablet_replica.cc:331] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.780314 29783 raft_consensus.cc:2241] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.780699 29783 raft_consensus.cc:2270] T 8d5844121c7e4b4c968a544c1b568e90 P 36d29e8094f34cacad7a80e1f2ddbed3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.782155 29783 tablet_replica.cc:331] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3: stopping tablet replica
I20250902 01:57:43.782564 29783 raft_consensus.cc:2241] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.782915 29783 raft_consensus.cc:2270] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 36d29e8094f34cacad7a80e1f2ddbed3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250902 01:57:43.790911 30471 consensus_peers.cc:489] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 -> Peer 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507): Couldn't send request to peer 36d29e8094f34cacad7a80e1f2ddbed3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:43.830168 29783 tablet_server.cc:195] TabletServer@127.29.21.193:0 shutdown complete.
I20250902 01:57:43.848302 29783 tablet_server.cc:178] TabletServer@127.29.21.194:0 shutting down...
I20250902 01:57:43.874207 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:43.874763 29783 tablet_replica.cc:331] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.875268 29783 raft_consensus.cc:2241] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 5 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.876008 29783 raft_consensus.cc:2270] T 777e53e58e4242408e7377ffe8ecb5a8 P 673abf5ce13140a7b854b7f50e0f6488 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.877553 29783 tablet_replica.cc:331] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.878014 29783 raft_consensus.cc:2241] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.878382 29783 raft_consensus.cc:2270] T ef2dd292e59f4e28b6349b96fdba1b04 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.879864 29783 tablet_replica.cc:331] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.880458 29783 raft_consensus.cc:2241] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.881305 29783 raft_consensus.cc:2270] T eb10f324c60e4436851b4abbcf2a503d P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.883074 29783 tablet_replica.cc:331] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.883770 29783 raft_consensus.cc:2241] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.884232 29783 raft_consensus.cc:2270] T c12b7b1fd6ef4af2a1a7e3917a553838 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250902 01:57:43.884959 30471 consensus_peers.cc:489] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 -> Peer 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507): Couldn't send request to peer 36d29e8094f34cacad7a80e1f2ddbed3. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:33507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:43.886032 29783 tablet_replica.cc:331] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.886494 29783 raft_consensus.cc:2241] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.886888 29783 raft_consensus.cc:2270] T 2746e7eebde8423f81678907aba6af92 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.888413 29783 tablet_replica.cc:331] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.888837 29783 raft_consensus.cc:2241] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.889216 29783 raft_consensus.cc:2270] T 30272acb192f4b15864059546a8019a3 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.890756 29783 tablet_replica.cc:331] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.891211 29783 raft_consensus.cc:2241] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.891628 29783 raft_consensus.cc:2270] T d404cc6e160a41458b944ce1811659cf P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.893088 29783 tablet_replica.cc:331] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.893543 29783 raft_consensus.cc:2241] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.894208 29783 raft_consensus.cc:2270] T 8d5844121c7e4b4c968a544c1b568e90 P 673abf5ce13140a7b854b7f50e0f6488 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.895804 29783 tablet_replica.cc:331] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.896234 29783 raft_consensus.cc:2241] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.896579 29783 raft_consensus.cc:2270] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.897960 29783 tablet_replica.cc:331] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488: stopping tablet replica
I20250902 01:57:43.898352 29783 raft_consensus.cc:2241] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.898703 29783 raft_consensus.cc:2270] T 43d6dabf0ff34265a47b8b39c2e84266 P 673abf5ce13140a7b854b7f50e0f6488 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.925429 29783 tablet_server.cc:195] TabletServer@127.29.21.194:0 shutdown complete.
I20250902 01:57:43.943604 29783 tablet_server.cc:178] TabletServer@127.29.21.195:0 shutting down...
I20250902 01:57:43.966321 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250902 01:57:43.966929 29783 tablet_replica.cc:331] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.967896 29783 raft_consensus.cc:2241] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.968444 29783 raft_consensus.cc:2270] T 30272acb192f4b15864059546a8019a3 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.970039 29783 tablet_replica.cc:331] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.970508 29783 raft_consensus.cc:2241] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.971174 29783 raft_consensus.cc:2270] T 2746e7eebde8423f81678907aba6af92 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.972771 29783 tablet_replica.cc:331] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.973224 29783 raft_consensus.cc:2241] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.973572 29783 raft_consensus.cc:2270] T 43d6dabf0ff34265a47b8b39c2e84266 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.975024 29783 tablet_replica.cc:331] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.975613 29783 raft_consensus.cc:2241] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 5 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.976023 29783 raft_consensus.cc:2270] T 777e53e58e4242408e7377ffe8ecb5a8 P 5437b7338b794400b09a7d407549227a [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.977598 29783 tablet_replica.cc:331] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.978050 29783 raft_consensus.cc:2241] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.978585 29783 raft_consensus.cc:2270] T 8d5844121c7e4b4c968a544c1b568e90 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Raft consensus is shut down!
W20250902 01:57:43.980015 30545 consensus_peers.cc:489] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a -> Peer 36d29e8094f34cacad7a80e1f2ddbed3 (127.29.21.193:33507): Couldn't send request to peer 36d29e8094f34cacad7a80e1f2ddbed3. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:33507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:43.980293 29783 tablet_replica.cc:331] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.980741 29783 raft_consensus.cc:2241] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.981410 29783 raft_consensus.cc:2270] T 2d12e1a2135c4ebeaf1adf6edb9aa07e P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.983018 29783 tablet_replica.cc:331] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.983495 29783 raft_consensus.cc:2241] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 2 LEADER]: Raft consensus shutting down.
I20250902 01:57:43.984205 29783 raft_consensus.cc:2270] T d404cc6e160a41458b944ce1811659cf P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.985706 29783 tablet_replica.cc:331] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.986142 29783 raft_consensus.cc:2241] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.986500 29783 raft_consensus.cc:2270] T eb10f324c60e4436851b4abbcf2a503d P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.987991 29783 tablet_replica.cc:331] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.988435 29783 raft_consensus.cc:2241] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.988763 29783 raft_consensus.cc:2270] T c12b7b1fd6ef4af2a1a7e3917a553838 P 5437b7338b794400b09a7d407549227a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:43.990226 29783 tablet_replica.cc:331] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a: stopping tablet replica
I20250902 01:57:43.990622 29783 raft_consensus.cc:2241] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:57:43.991012 29783 raft_consensus.cc:2270] T ef2dd292e59f4e28b6349b96fdba1b04 P 5437b7338b794400b09a7d407549227a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:44.020032 29783 tablet_server.cc:195] TabletServer@127.29.21.195:0 shutdown complete.
I20250902 01:57:44.038590 29783 master.cc:561] Master@127.29.21.254:43255 shutting down...
I20250902 01:57:44.058396 29783 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 1 LEADER]: Raft consensus shutting down.
I20250902 01:57:44.058877 29783 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:57:44.059158 29783 tablet_replica.cc:331] T 00000000000000000000000000000000 P 87409d5661214b34a0d19f6b74641dc4: stopping tablet replica
I20250902 01:57:44.077960 29783 master.cc:583] Master@127.29.21.254:43255 shutdown complete.
36d29e8094f34cacad7a80e1f2ddbed3 5
5437b7338b794400b09a7d407549227a 1
673abf5ce13140a7b854b7f50e0f6488 4
36d29e8094f34cacad7a80e1f2ddbed3 3
5437b7338b794400b09a7d407549227a 3
673abf5ce13140a7b854b7f50e0f6488 4
36d29e8094f34cacad7a80e1f2ddbed3 8
5437b7338b794400b09a7d407549227a 1
673abf5ce13140a7b854b7f50e0f6488 1
36d29e8094f34cacad7a80e1f2ddbed3 4
5437b7338b794400b09a7d407549227a 3
673abf5ce13140a7b854b7f50e0f6488 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28480 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250902 01:57:44.115258 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:45755
I20250902 01:57:44.116348 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:44.121677 30799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.121731 30800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.122776 30802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:44.123342 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:44.124135 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:44.124315 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:44.124423 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778264124412 us; error 0 us; skew 500 ppm
I20250902 01:57:44.124841 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:44.130995 29783 webserver.cc:480] Webserver started at http://127.29.21.254:44581/ using document root <none> and password file <none>
I20250902 01:57:44.131417 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:44.131834 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:44.132058 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:44.133064 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/master-0-root/instance:
uuid: "7a3020b0c4d54495a77a3aff71003d3b"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.137032 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250902 01:57:44.139983 30807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.140616 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250902 01:57:44.140854 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/master-0-root
uuid: "7a3020b0c4d54495a77a3aff71003d3b"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.141078 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:44.156862 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:44.157881 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:44.192756 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:45755
I20250902 01:57:44.192838 30858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.254:45755 every 8 connection(s)
I20250902 01:57:44.196374 30859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.205345 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Bootstrap starting.
I20250902 01:57:44.209481 30859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:44.213179 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: No bootstrap required, opened a new log
I20250902 01:57:44.215032 30859 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER }
I20250902 01:57:44.215401 30859 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:44.215655 30859 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3020b0c4d54495a77a3aff71003d3b, State: Initialized, Role: FOLLOWER
I20250902 01:57:44.216194 30859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER }
I20250902 01:57:44.216645 30859 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250902 01:57:44.216852 30859 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250902 01:57:44.217106 30859 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:44.220805 30859 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER }
I20250902 01:57:44.221340 30859 leader_election.cc:304] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3020b0c4d54495a77a3aff71003d3b; no voters:
I20250902 01:57:44.222326 30859 leader_election.cc:290] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250902 01:57:44.222649 30862 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:44.223868 30862 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [term 1 LEADER]: Becoming Leader. State: Replica: 7a3020b0c4d54495a77a3aff71003d3b, State: Running, Role: LEADER
I20250902 01:57:44.224416 30862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER }
I20250902 01:57:44.224995 30859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [sys.catalog]: configured and running, proceeding with master startup.
I20250902 01:57:44.228536 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3020b0c4d54495a77a3aff71003d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER } }
I20250902 01:57:44.229220 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [sys.catalog]: This master's current role is: LEADER
I20250902 01:57:44.229209 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3020b0c4d54495a77a3aff71003d3b. Latest consensus state: current_term: 1 leader_uuid: "7a3020b0c4d54495a77a3aff71003d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3020b0c4d54495a77a3aff71003d3b" member_type: VOTER } }
I20250902 01:57:44.230013 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b [sys.catalog]: This master's current role is: LEADER
I20250902 01:57:44.233018 30869 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250902 01:57:44.240653 30869 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250902 01:57:44.241557 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250902 01:57:44.248816 30869 catalog_manager.cc:1349] Generated new cluster ID: 7e96cf1f10a942e99b1521b844bb7309
I20250902 01:57:44.249013 30869 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250902 01:57:44.295096 30869 catalog_manager.cc:1372] Generated new certificate authority record
I20250902 01:57:44.296687 30869 catalog_manager.cc:1506] Loading token signing keys...
I20250902 01:57:44.311022 30869 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Generated new TSK 0
I20250902 01:57:44.311651 30869 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250902 01:57:44.371912 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:44.377254 30880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.378400 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.379750 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:44.379984 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:44.380858 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:44.381047 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:44.381199 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778264381180 us; error 0 us; skew 500 ppm
I20250902 01:57:44.381708 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:44.383791 29783 webserver.cc:480] Webserver started at http://127.29.21.193:44113/ using document root <none> and password file <none>
I20250902 01:57:44.384229 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:44.384402 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:44.384653 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:44.385632 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/instance:
uuid: "4372b4c4c866431faa1bf3e91fd47a88"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.389770 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250902 01:57:44.392714 30888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.393335 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250902 01:57:44.393599 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root
uuid: "4372b4c4c866431faa1bf3e91fd47a88"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.393843 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:44.409807 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:44.410907 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:44.412261 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:44.414239 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:44.414427 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.414662 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:44.414813 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.452054 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:44457
I20250902 01:57:44.452139 30950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.193:44457 every 8 connection(s)
I20250902 01:57:44.456329 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:44.463459 30955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.465544 30956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.467296 30958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:44.467682 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:44.468490 30951 heartbeater.cc:344] Connected to a master server at 127.29.21.254:45755
I20250902 01:57:44.468688 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250902 01:57:44.468813 30951 heartbeater.cc:461] Registering TS with master...
W20250902 01:57:44.468930 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:44.469151 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778264469136 us; error 0 us; skew 500 ppm
I20250902 01:57:44.469556 30951 heartbeater.cc:507] Master 127.29.21.254:45755 requested a full tablet report, sending...
I20250902 01:57:44.469653 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:44.471709 30824 ts_manager.cc:194] Registered new tserver with Master: 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:57:44.472338 29783 webserver.cc:480] Webserver started at http://127.29.21.194:35111/ using document root <none> and password file <none>
I20250902 01:57:44.472785 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:44.472959 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:44.473191 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:44.473420 30824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44296
I20250902 01:57:44.474315 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/instance:
uuid: "975ecd397f5146409baac87e56a302e9"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.478408 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250902 01:57:44.481340 30963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.482066 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250902 01:57:44.482304 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root
uuid: "975ecd397f5146409baac87e56a302e9"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.482548 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:44.497726 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:44.498700 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:44.500007 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:44.501977 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:44.502161 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.502393 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:44.502537 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.538816 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:46439
I20250902 01:57:44.538906 31025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.194:46439 every 8 connection(s)
I20250902 01:57:44.543260 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:44.549822 31030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.549537 31029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:44.552500 31032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:44.553875 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:44.554028 31026 heartbeater.cc:344] Connected to a master server at 127.29.21.254:45755
I20250902 01:57:44.554565 31026 heartbeater.cc:461] Registering TS with master...
I20250902 01:57:44.554867 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:44.555104 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:44.555253 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778264555236 us; error 0 us; skew 500 ppm
I20250902 01:57:44.555271 31026 heartbeater.cc:507] Master 127.29.21.254:45755 requested a full tablet report, sending...
I20250902 01:57:44.555830 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:44.557180 30824 ts_manager.cc:194] Registered new tserver with Master: 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:44.557998 29783 webserver.cc:480] Webserver started at http://127.29.21.195:43447/ using document root <none> and password file <none>
I20250902 01:57:44.558463 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:44.558461 30824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44308
I20250902 01:57:44.558782 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:44.559059 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:44.560253 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/instance:
uuid: "088424002a894fd5bcc11a06af111445"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.564150 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250902 01:57:44.566859 31037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.567469 29783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250902 01:57:44.567736 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root
uuid: "088424002a894fd5bcc11a06af111445"
format_stamp: "Formatted at 2025-09-02 01:57:44 on dist-test-slave-g0c9"
I20250902 01:57:44.567967 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:44.593411 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:44.594563 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:44.596045 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:44.598299 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:44.598495 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.598717 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:44.598867 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:44.635944 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:40531
I20250902 01:57:44.636057 31099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.195:40531 every 8 connection(s)
I20250902 01:57:44.648100 31100 heartbeater.cc:344] Connected to a master server at 127.29.21.254:45755
I20250902 01:57:44.648448 31100 heartbeater.cc:461] Registering TS with master...
I20250902 01:57:44.649154 31100 heartbeater.cc:507] Master 127.29.21.254:45755 requested a full tablet report, sending...
I20250902 01:57:44.650753 30824 ts_manager.cc:194] Registered new tserver with Master: 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:44.651019 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012111505s
I20250902 01:57:44.652432 30824 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44316
I20250902 01:57:44.653831 29783 test_util.cc:276] Using random seed: -798363670
I20250902 01:57:44.674942 30824 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44320:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250902 01:57:44.680598 30824 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250902 01:57:44.866478 30991 tablet_service.cc:1468] Processing CreateTablet for tablet eecf760e0123488fb797b4457e0888d7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250902 01:57:44.867858 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecf760e0123488fb797b4457e0888d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.867969 30987 tablet_service.cc:1468] Processing CreateTablet for tablet 6d30dc90c304472ab905d3af8400b8d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250902 01:57:44.869355 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30dc90c304472ab905d3af8400b8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.869763 30989 tablet_service.cc:1468] Processing CreateTablet for tablet 02cd1389058944c1adc90f23d0fadb21 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250902 01:57:44.871110 30989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cd1389058944c1adc90f23d0fadb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.871735 31065 tablet_service.cc:1468] Processing CreateTablet for tablet eecf760e0123488fb797b4457e0888d7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250902 01:57:44.872953 31065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecf760e0123488fb797b4457e0888d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.872723 30990 tablet_service.cc:1468] Processing CreateTablet for tablet 26ad5d89b038487b8bbc776913d5a215 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250902 01:57:44.872766 31064 tablet_service.cc:1468] Processing CreateTablet for tablet 26ad5d89b038487b8bbc776913d5a215 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250902 01:57:44.873998 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ad5d89b038487b8bbc776913d5a215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.874063 31064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ad5d89b038487b8bbc776913d5a215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.868068 30988 tablet_service.cc:1468] Processing CreateTablet for tablet cb3bd5884ec54e9da4138c559675467e (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250902 01:57:44.876297 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3bd5884ec54e9da4138c559675467e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.876730 31063 tablet_service.cc:1468] Processing CreateTablet for tablet 02cd1389058944c1adc90f23d0fadb21 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250902 01:57:44.877923 31063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cd1389058944c1adc90f23d0fadb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.880575 31062 tablet_service.cc:1468] Processing CreateTablet for tablet cb3bd5884ec54e9da4138c559675467e (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250902 01:57:44.881891 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3bd5884ec54e9da4138c559675467e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.886561 31061 tablet_service.cc:1468] Processing CreateTablet for tablet 6d30dc90c304472ab905d3af8400b8d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250902 01:57:44.887881 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30dc90c304472ab905d3af8400b8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.890740 30984 tablet_service.cc:1468] Processing CreateTablet for tablet 151f832cfd4e42d3a547a7cef6bb995f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250902 01:57:44.891254 31060 tablet_service.cc:1468] Processing CreateTablet for tablet 22b1ce0ddf934a6baa56f3ee89e05309 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250902 01:57:44.892114 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151f832cfd4e42d3a547a7cef6bb995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.892675 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b1ce0ddf934a6baa56f3ee89e05309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.894496 30985 tablet_service.cc:1468] Processing CreateTablet for tablet 8f5d81000607410da8880ef8e5517680 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250902 01:57:44.895116 31059 tablet_service.cc:1468] Processing CreateTablet for tablet 8f5d81000607410da8880ef8e5517680 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250902 01:57:44.896950 30986 tablet_service.cc:1468] Processing CreateTablet for tablet 22b1ce0ddf934a6baa56f3ee89e05309 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250902 01:57:44.896731 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5d81000607410da8880ef8e5517680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.899341 30983 tablet_service.cc:1468] Processing CreateTablet for tablet 818cd33c83fe427a981bf0c807af5b14 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250902 01:57:44.900597 30985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5d81000607410da8880ef8e5517680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.904198 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd33c83fe427a981bf0c807af5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.908054 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b1ce0ddf934a6baa56f3ee89e05309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.911590 31058 tablet_service.cc:1468] Processing CreateTablet for tablet 151f832cfd4e42d3a547a7cef6bb995f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250902 01:57:44.912962 31058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151f832cfd4e42d3a547a7cef6bb995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.913120 30987 tablet_service.cc:1468] Processing CreateTablet for tablet 9c9ad89315794238b432653557334854 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250902 01:57:44.914239 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9ad89315794238b432653557334854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.917302 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 50a1d33046ce49cd891b784fbe726d64 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250902 01:57:44.917498 31057 tablet_service.cc:1468] Processing CreateTablet for tablet 818cd33c83fe427a981bf0c807af5b14 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250902 01:57:44.918503 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50a1d33046ce49cd891b784fbe726d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.918996 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd33c83fe427a981bf0c807af5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.921048 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 9c9ad89315794238b432653557334854 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250902 01:57:44.922302 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9ad89315794238b432653557334854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.932578 30991 tablet_service.cc:1468] Processing CreateTablet for tablet 805e9f7636dc47059b5af8c090e5ac0a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250902 01:57:44.933478 31115 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:44.947486 31115 tablet_bootstrap.cc:654] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:44.972440 31062 tablet_service.cc:1468] Processing CreateTablet for tablet 50a1d33046ce49cd891b784fbe726d64 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250902 01:57:44.974377 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50a1d33046ce49cd891b784fbe726d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.983376 31057 tablet_service.cc:1468] Processing CreateTablet for tablet 0c93c2499f7743579a074e5f2646ad2d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250902 01:57:44.984740 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c93c2499f7743579a074e5f2646ad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.987784 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805e9f7636dc47059b5af8c090e5ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:44.989344 31056 tablet_service.cc:1468] Processing CreateTablet for tablet dad02ef06fef44c5b5ecc42134c43091 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250902 01:57:44.992901 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad02ef06fef44c5b5ecc42134c43091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.002100 30914 tablet_service.cc:1468] Processing CreateTablet for tablet 02cd1389058944c1adc90f23d0fadb21 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250902 01:57:45.003394 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cd1389058944c1adc90f23d0fadb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.005765 30916 tablet_service.cc:1468] Processing CreateTablet for tablet eecf760e0123488fb797b4457e0888d7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250902 01:57:45.005807 30989 tablet_service.cc:1468] Processing CreateTablet for tablet dad02ef06fef44c5b5ecc42134c43091 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250902 01:57:45.007112 30916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecf760e0123488fb797b4457e0888d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.008677 30913 tablet_service.cc:1468] Processing CreateTablet for tablet cb3bd5884ec54e9da4138c559675467e (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250902 01:57:45.009883 30913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3bd5884ec54e9da4138c559675467e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.012756 30912 tablet_service.cc:1468] Processing CreateTablet for tablet 6d30dc90c304472ab905d3af8400b8d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250902 01:57:45.013171 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 0c93c2499f7743579a074e5f2646ad2d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250902 01:57:45.014588 30983 tablet_service.cc:1468] Processing CreateTablet for tablet 6c9675a8370e4d4a9f95b3c568abe972 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250902 01:57:45.014703 30911 tablet_service.cc:1468] Processing CreateTablet for tablet 22b1ce0ddf934a6baa56f3ee89e05309 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250902 01:57:45.015875 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b1ce0ddf934a6baa56f3ee89e05309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.020154 31058 tablet_service.cc:1468] Processing CreateTablet for tablet 805e9f7636dc47059b5af8c090e5ac0a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250902 01:57:45.020620 30910 tablet_service.cc:1468] Processing CreateTablet for tablet 8f5d81000607410da8880ef8e5517680 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250902 01:57:45.021432 31058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805e9f7636dc47059b5af8c090e5ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.021869 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5d81000607410da8880ef8e5517680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.023713 30909 tablet_service.cc:1468] Processing CreateTablet for tablet 151f832cfd4e42d3a547a7cef6bb995f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250902 01:57:45.025696 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151f832cfd4e42d3a547a7cef6bb995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.026315 30908 tablet_service.cc:1468] Processing CreateTablet for tablet 818cd33c83fe427a981bf0c807af5b14 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250902 01:57:45.027495 30908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd33c83fe427a981bf0c807af5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.035967 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30dc90c304472ab905d3af8400b8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.036295 30985 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3db1aaea8d4a3eba2de840361e0849 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250902 01:57:45.002100 30915 tablet_service.cc:1468] Processing CreateTablet for tablet 26ad5d89b038487b8bbc776913d5a215 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250902 01:57:45.040809 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ad5d89b038487b8bbc776913d5a215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.024196 30987 tablet_service.cc:1468] Processing CreateTablet for tablet 24426e817ea94ac5a7014c3457ea087c (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250902 01:57:45.043375 31059 tablet_service.cc:1468] Processing CreateTablet for tablet 24426e817ea94ac5a7014c3457ea087c (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250902 01:57:45.042850 30990 tablet_service.cc:1468] Processing CreateTablet for tablet 92f16b8a15144fcf8807608bdee71123 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250902 01:57:45.044544 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24426e817ea94ac5a7014c3457ea087c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.028700 30907 tablet_service.cc:1468] Processing CreateTablet for tablet 9c9ad89315794238b432653557334854 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250902 01:57:45.045840 30907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c9ad89315794238b432653557334854. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.043375 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 6c9675a8370e4d4a9f95b3c568abe972 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250902 01:57:45.047981 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9675a8370e4d4a9f95b3c568abe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.062711 31064 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3db1aaea8d4a3eba2de840361e0849 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250902 01:57:45.064327 31064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3db1aaea8d4a3eba2de840361e0849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.007138 30989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad02ef06fef44c5b5ecc42134c43091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.084906 31061 tablet_service.cc:1468] Processing CreateTablet for tablet 92f16b8a15144fcf8807608bdee71123 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250902 01:57:45.090147 31115 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:45.090476 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f16b8a15144fcf8807608bdee71123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.090785 31115 ts_tablet_manager.cc:1397] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.171s user 0.009s sys 0.005s
I20250902 01:57:45.094349 31060 tablet_service.cc:1468] Processing CreateTablet for tablet e12bca11fa644c12b4c6f0658fa07084 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250902 01:57:45.094280 30988 tablet_service.cc:1468] Processing CreateTablet for tablet e12bca11fa644c12b4c6f0658fa07084 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250902 01:57:45.098423 31116 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:45.105613 31062 tablet_service.cc:1468] Processing CreateTablet for tablet 568a81baa7ef422eb8125c6e8bb9813b (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250902 01:57:45.107266 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12bca11fa644c12b4c6f0658fa07084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.110297 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9675a8370e4d4a9f95b3c568abe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.110940 30985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3db1aaea8d4a3eba2de840361e0849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.115841 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f16b8a15144fcf8807608bdee71123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.116801 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24426e817ea94ac5a7014c3457ea087c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.116943 31063 tablet_service.cc:1468] Processing CreateTablet for tablet a61d1c51a3034d52aa9a91fea8525fc7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250902 01:57:45.117652 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12bca11fa644c12b4c6f0658fa07084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.118098 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568a81baa7ef422eb8125c6e8bb9813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.122504 30986 tablet_service.cc:1468] Processing CreateTablet for tablet 568a81baa7ef422eb8125c6e8bb9813b (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250902 01:57:45.123378 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c93c2499f7743579a074e5f2646ad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.123755 31063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61d1c51a3034d52aa9a91fea8525fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.143404 30991 tablet_service.cc:1468] Processing CreateTablet for tablet a61d1c51a3034d52aa9a91fea8525fc7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250902 01:57:45.145540 30984 tablet_service.cc:1468] Processing CreateTablet for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250902 01:57:45.138232 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250902 01:57:45.157182 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e0a90ceb7a486aa3a93a73f087dfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.163763 30908 tablet_service.cc:1468] Processing CreateTablet for tablet 50a1d33046ce49cd891b784fbe726d64 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250902 01:57:45.165068 30908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50a1d33046ce49cd891b784fbe726d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.157437 31115 raft_consensus.cc:357] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.173100 31060 tablet_service.cc:1468] Processing CreateTablet for tablet aa4b903a93464946ae238ce634b483d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250902 01:57:45.174293 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4b903a93464946ae238ce634b483d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.174525 30912 tablet_service.cc:1468] Processing CreateTablet for tablet dad02ef06fef44c5b5ecc42134c43091 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250902 01:57:45.168187 31115 raft_consensus.cc:383] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.175705 31115 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.176352 31115 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.177632 30915 tablet_service.cc:1468] Processing CreateTablet for tablet 0c93c2499f7743579a074e5f2646ad2d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250902 01:57:45.180361 31065 tablet_service.cc:1468] Processing CreateTablet for tablet 39363bee7fbd45fdaa81366e5da31356 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250902 01:57:45.169126 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568a81baa7ef422eb8125c6e8bb9813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.179981 30916 tablet_service.cc:1468] Processing CreateTablet for tablet 805e9f7636dc47059b5af8c090e5ac0a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250902 01:57:45.194537 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dad02ef06fef44c5b5ecc42134c43091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.185560 30985 tablet_service.cc:1468] Processing CreateTablet for tablet 39363bee7fbd45fdaa81366e5da31356 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250902 01:57:45.187211 30907 tablet_service.cc:1468] Processing CreateTablet for tablet 24426e817ea94ac5a7014c3457ea087c (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250902 01:57:45.197096 30907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24426e817ea94ac5a7014c3457ea087c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.187748 31065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39363bee7fbd45fdaa81366e5da31356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.187212 30909 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3db1aaea8d4a3eba2de840361e0849 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250902 01:57:45.204784 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3db1aaea8d4a3eba2de840361e0849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.204914 30914 tablet_service.cc:1468] Processing CreateTablet for tablet e12bca11fa644c12b4c6f0658fa07084 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250902 01:57:45.206120 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12bca11fa644c12b4c6f0658fa07084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.206423 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61d1c51a3034d52aa9a91fea8525fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.208128 30985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39363bee7fbd45fdaa81366e5da31356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.186170 30911 tablet_service.cc:1468] Processing CreateTablet for tablet 6c9675a8370e4d4a9f95b3c568abe972 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250902 01:57:45.190897 30910 tablet_service.cc:1468] Processing CreateTablet for tablet 92f16b8a15144fcf8807608bdee71123 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250902 01:57:45.212244 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9675a8370e4d4a9f95b3c568abe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.193918 30988 tablet_service.cc:1468] Processing CreateTablet for tablet aa4b903a93464946ae238ce634b483d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250902 01:57:45.213666 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e0a90ceb7a486aa3a93a73f087dfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.213961 31063 tablet_service.cc:1468] Processing CreateTablet for tablet d7f66bcc25964883baf67f48889ec766 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250902 01:57:45.215148 31063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f66bcc25964883baf67f48889ec766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.215826 31061 tablet_service.cc:1468] Processing CreateTablet for tablet 69f27fd54f16465fa39cc86d8d6c07bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250902 01:57:45.217341 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f27fd54f16465fa39cc86d8d6c07bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.224283 30913 tablet_service.cc:1468] Processing CreateTablet for tablet 568a81baa7ef422eb8125c6e8bb9813b (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250902 01:57:45.190897 30987 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2dd8f5f814a7d8b48a74020055750 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250902 01:57:45.181627 31116 tablet_bootstrap.cc:654] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.226383 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2dd8f5f814a7d8b48a74020055750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.232527 31062 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf39ccba06f4a0998fb41dd68a0702a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250902 01:57:45.233590 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf39ccba06f4a0998fb41dd68a0702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.233474 31059 tablet_service.cc:1468] Processing CreateTablet for tablet 2925593648194df5b7741d448a047401 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250902 01:57:45.234694 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925593648194df5b7741d448a047401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.235661 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4b903a93464946ae238ce634b483d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.237689 31060 tablet_service.cc:1468] Processing CreateTablet for tablet 4120ecbc46e34f93bb86004798318c08 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250902 01:57:45.238976 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120ecbc46e34f93bb86004798318c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.253719 31118 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:45.257692 31057 tablet_service.cc:1468] Processing CreateTablet for tablet 331364bf121c480483b11ff32993e1ec (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250902 01:57:45.258896 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331364bf121c480483b11ff32993e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.261993 30983 tablet_service.cc:1468] Processing CreateTablet for tablet d7f66bcc25964883baf67f48889ec766 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250902 01:57:45.263211 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f66bcc25964883baf67f48889ec766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.273520 30991 tablet_service.cc:1468] Processing CreateTablet for tablet 69f27fd54f16465fa39cc86d8d6c07bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250902 01:57:45.274735 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f27fd54f16465fa39cc86d8d6c07bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.280683 31115 ts_tablet_manager.cc:1428] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.190s user 0.016s sys 0.004s
I20250902 01:57:45.280388 31061 tablet_service.cc:1468] Processing CreateTablet for tablet 21e596cb1bea4b1c9015d9b8d9a188b6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250902 01:57:45.281394 31026 heartbeater.cc:499] Master 127.29.21.254:45755 was elected leader, sending a full tablet report...
I20250902 01:57:45.281652 31115 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:45.281785 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e596cb1bea4b1c9015d9b8d9a188b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.292447 31059 tablet_service.cc:1468] Processing CreateTablet for tablet ade302bdd69a4fa7876c90a1fbd87b68 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20250902 01:57:45.292783 31027 tablet.cc:2378] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250902 01:57:45.294260 31060 tablet_service.cc:1468] Processing CreateTablet for tablet 9685199b6c7b42f391ab48fdf86d71ad (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250902 01:57:45.294499 31064 tablet_service.cc:1468] Processing CreateTablet for tablet aeb7c457a3f741b5b4d057f06363c614 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250902 01:57:45.296211 31115 tablet_bootstrap.cc:654] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.303077 31116 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:45.303996 31116 ts_tablet_manager.cc:1397] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.206s user 0.017s sys 0.033s
I20250902 01:57:45.307144 31116 raft_consensus.cc:357] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.307946 31116 raft_consensus.cc:383] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.307747 31056 tablet_service.cc:1468] Processing CreateTablet for tablet f0f24bac96dc42dba9adc1f6679c0d48 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250902 01:57:45.308252 31116 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.308830 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade302bdd69a4fa7876c90a1fbd87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.309618 31116 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.315999 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685199b6c7b42f391ab48fdf86d71ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.319501 31116 ts_tablet_manager.cc:1428] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20250902 01:57:45.320153 31100 heartbeater.cc:499] Master 127.29.21.254:45755 was elected leader, sending a full tablet report...
W20250902 01:57:45.351769 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:45.364408 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f16b8a15144fcf8807608bdee71123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.376155 30913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568a81baa7ef422eb8125c6e8bb9813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.377269 30916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805e9f7636dc47059b5af8c090e5ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.384033 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c93c2499f7743579a074e5f2646ad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.288208 30986 tablet_service.cc:1468] Processing CreateTablet for tablet 4120ecbc46e34f93bb86004798318c08 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250902 01:57:45.410786 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120ecbc46e34f93bb86004798318c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.415719 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 331364bf121c480483b11ff32993e1ec (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250902 01:57:45.418068 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331364bf121c480483b11ff32993e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.418443 30987 tablet_service.cc:1468] Processing CreateTablet for tablet 21e596cb1bea4b1c9015d9b8d9a188b6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250902 01:57:45.420663 30989 tablet_service.cc:1468] Processing CreateTablet for tablet 2925593648194df5b7741d448a047401 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250902 01:57:45.297585 30990 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf39ccba06f4a0998fb41dd68a0702a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250902 01:57:45.439275 31064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb7c457a3f741b5b4d057f06363c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.441917 31115 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:45.446861 31061 tablet_service.cc:1468] Processing CreateTablet for tablet 744f11902df74dea8914fdd4a992259d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250902 01:57:45.442818 31120 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:45.447921 31120 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.449390 31062 tablet_service.cc:1468] Processing CreateTablet for tablet 416020230983423c9f843070a7893658 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250902 01:57:45.452174 31063 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2dd8f5f814a7d8b48a74020055750 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250902 01:57:45.445178 30989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925593648194df5b7741d448a047401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.442817 31123 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:45.447059 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f24bac96dc42dba9adc1f6679c0d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.456343 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e596cb1bea4b1c9015d9b8d9a188b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.456074 31123 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.457702 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf39ccba06f4a0998fb41dd68a0702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.458463 31116 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:45.321547 31065 tablet_service.cc:1468] Processing CreateTablet for tablet 59bc91ec50764c7bb821a4288bea8d16 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250902 01:57:45.462357 31065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59bc91ec50764c7bb821a4288bea8d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.464277 31063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2dd8f5f814a7d8b48a74020055750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.455246 31120 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:57:45.442147 31058 tablet_service.cc:1468] Processing CreateTablet for tablet de2e42094ddd41f99d12a4972ee07b9f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250902 01:57:45.474339 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416020230983423c9f843070a7893658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.475163 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744f11902df74dea8914fdd4a992259d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.476246 30951 heartbeater.cc:499] Master 127.29.21.254:45755 was elected leader, sending a full tablet report...
I20250902 01:57:45.477164 31060 tablet_service.cc:1468] Processing CreateTablet for tablet 0692c5029b5e4ab088324d038ff964be (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250902 01:57:45.480484 30988 tablet_service.cc:1468] Processing CreateTablet for tablet 9685199b6c7b42f391ab48fdf86d71ad (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250902 01:57:45.471769 31115 ts_tablet_manager.cc:1397] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.190s user 0.011s sys 0.006s
I20250902 01:57:45.481691 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685199b6c7b42f391ab48fdf86d71ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.493846 31115 raft_consensus.cc:357] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.504387 31115 raft_consensus.cc:383] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.504776 31115 raft_consensus.cc:738] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.495067 31058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2e42094ddd41f99d12a4972ee07b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.505810 31057 tablet_service.cc:1468] Processing CreateTablet for tablet b03cb35549a346d28baa21d708d665e2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250902 01:57:45.506201 30914 tablet_service.cc:1468] Processing CreateTablet for tablet a61d1c51a3034d52aa9a91fea8525fc7 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250902 01:57:45.507140 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03cb35549a346d28baa21d708d665e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.507308 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a61d1c51a3034d52aa9a91fea8525fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.511976 31115 consensus_queue.cc:260] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.532858 31115 ts_tablet_manager.cc:1428] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.052s user 0.002s sys 0.005s
I20250902 01:57:45.536830 30984 tablet_service.cc:1468] Processing CreateTablet for tablet aeb7c457a3f741b5b4d057f06363c614 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250902 01:57:45.538256 31115 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:45.539830 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0692c5029b5e4ab088324d038ff964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.544062 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb7c457a3f741b5b4d057f06363c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.545984 30991 tablet_service.cc:1468] Processing CreateTablet for tablet f0f24bac96dc42dba9adc1f6679c0d48 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250902 01:57:45.549582 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f24bac96dc42dba9adc1f6679c0d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.553889 31123 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:57:45.558988 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 59bc91ec50764c7bb821a4288bea8d16 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250902 01:57:45.539441 31118 tablet_bootstrap.cc:654] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.567042 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 2e3ca4f287b94f60b51ef1768df7bb70 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250902 01:57:45.568295 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3ca4f287b94f60b51ef1768df7bb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.568492 30912 tablet_service.cc:1468] Processing CreateTablet for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250902 01:57:45.569546 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e0a90ceb7a486aa3a93a73f087dfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.585111 30911 tablet_service.cc:1468] Processing CreateTablet for tablet 39363bee7fbd45fdaa81366e5da31356 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250902 01:57:45.586372 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39363bee7fbd45fdaa81366e5da31356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.592623 31059 tablet_service.cc:1468] Processing CreateTablet for tablet 349593d2d3574cd6ab281c105756f435 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250902 01:57:45.593158 30908 tablet_service.cc:1468] Processing CreateTablet for tablet aa4b903a93464946ae238ce634b483d0 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250902 01:57:45.593777 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349593d2d3574cd6ab281c105756f435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.594355 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 63fa3c697d3e40afb9a01578cbe72700 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250902 01:57:45.595487 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fa3c697d3e40afb9a01578cbe72700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.595639 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59bc91ec50764c7bb821a4288bea8d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.599663 30915 tablet_service.cc:1468] Processing CreateTablet for tablet d7f66bcc25964883baf67f48889ec766 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250902 01:57:45.602528 31057 tablet_service.cc:1468] Processing CreateTablet for tablet 38649bfc334c4db6921eb3b12130e10d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250902 01:57:45.603976 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38649bfc334c4db6921eb3b12130e10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.610486 31056 tablet_service.cc:1468] Processing CreateTablet for tablet a2d2c9c23b6145c9b6601f7e78f1f43a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250902 01:57:45.611697 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d2c9c23b6145c9b6601f7e78f1f43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.615636 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:45.617415 30965 leader_election.cc:343] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.618701 30916 tablet_service.cc:1468] Processing CreateTablet for tablet 69f27fd54f16465fa39cc86d8d6c07bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250902 01:57:45.622426 31060 tablet_service.cc:1468] Processing CreateTablet for tablet f160859afd5945949c030026bfe3ac61 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250902 01:57:45.624434 31060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f160859afd5945949c030026bfe3ac61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.624083 31058 tablet_service.cc:1468] Processing CreateTablet for tablet 558db87de6fa4361bf0ca91e27d34943 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250902 01:57:45.594368 30908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4b903a93464946ae238ce634b483d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.625545 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f66bcc25964883baf67f48889ec766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.626370 31056 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba5aa5dce214071a172ea365440f0dc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250902 01:57:45.628962 30907 tablet_service.cc:1468] Processing CreateTablet for tablet 4120ecbc46e34f93bb86004798318c08 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250902 01:57:45.630082 30907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4120ecbc46e34f93bb86004798318c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.543239 31061 tablet_service.cc:1468] Processing CreateTablet for tablet d920df318de242f797c216f655b63218 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250902 01:57:45.633440 30910 tablet_service.cc:1468] Processing CreateTablet for tablet 5cf39ccba06f4a0998fb41dd68a0702a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250902 01:57:45.644215 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf39ccba06f4a0998fb41dd68a0702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.644851 30983 tablet_service.cc:1468] Processing CreateTablet for tablet 744f11902df74dea8914fdd4a992259d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250902 01:57:45.639216 31115 tablet_bootstrap.cc:654] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.650432 30916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f27fd54f16465fa39cc86d8d6c07bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.653190 30989 tablet_service.cc:1468] Processing CreateTablet for tablet 416020230983423c9f843070a7893658 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250902 01:57:45.654428 30989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416020230983423c9f843070a7893658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.660702 31116 tablet_bootstrap.cc:654] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.661852 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744f11902df74dea8914fdd4a992259d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.667667 30985 tablet_service.cc:1468] Processing CreateTablet for tablet ade302bdd69a4fa7876c90a1fbd87b68 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250902 01:57:45.669756 30985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade302bdd69a4fa7876c90a1fbd87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.668855 30986 tablet_service.cc:1468] Processing CreateTablet for tablet de2e42094ddd41f99d12a4972ee07b9f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250902 01:57:45.672302 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2e42094ddd41f99d12a4972ee07b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.668535 31065 tablet_service.cc:1468] Processing CreateTablet for tablet 543054d6241649d59bb40156217a73bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250902 01:57:45.684289 30987 tablet_service.cc:1468] Processing CreateTablet for tablet b03cb35549a346d28baa21d708d665e2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250902 01:57:45.685566 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03cb35549a346d28baa21d708d665e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.692714 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:45.693431 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:45.694053 31001 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
W20250902 01:57:45.694237 31039 leader_election.cc:343] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.695402 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:45.695902 30988 tablet_service.cc:1468] Processing CreateTablet for tablet 0692c5029b5e4ab088324d038ff964be (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250902 01:57:45.696233 30909 tablet_service.cc:1468] Processing CreateTablet for tablet 2925593648194df5b7741d448a047401 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250902 01:57:45.696784 31123 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:45.697161 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0692c5029b5e4ab088324d038ff964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.697127 31123 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:45.697399 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925593648194df5b7741d448a047401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.697588 31123 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:45.705258 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:45.705557 31123 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.706140 30991 tablet_service.cc:1468] Processing CreateTablet for tablet d920df318de242f797c216f655b63218 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250902 01:57:45.709115 31075 raft_consensus.cc:2391] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:45.710901 31123 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:57:45.711470 31057 tablet_service.cc:1468] Processing CreateTablet for tablet ede06f18c6574c57857475151e3767d2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250902 01:57:45.712033 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:45.712657 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:45.713150 31001 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:45.713253 31039 leader_election.cc:343] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.710659 30991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d920df318de242f797c216f655b63218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.720530 30990 tablet_service.cc:1468] Processing CreateTablet for tablet 2e3ca4f287b94f60b51ef1768df7bb70 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250902 01:57:45.728226 31115 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:45.729981 30967 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:45.730880 31057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede06f18c6574c57857475151e3767d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.737529 30911 tablet_service.cc:1468] Processing CreateTablet for tablet 21e596cb1bea4b1c9015d9b8d9a188b6 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250902 01:57:45.741564 31065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543054d6241649d59bb40156217a73bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.742313 31058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 558db87de6fa4361bf0ca91e27d34943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.747596 31115 ts_tablet_manager.cc:1397] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.210s user 0.012s sys 0.012s
I20250902 01:57:45.747843 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 349593d2d3574cd6ab281c105756f435 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250902 01:57:45.748713 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e596cb1bea4b1c9015d9b8d9a188b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.749083 31001 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:45.749619 31120 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:45.751006 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:45.751801 31061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d920df318de242f797c216f655b63218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.752285 31123 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:45.753134 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba5aa5dce214071a172ea365440f0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.719599 30910 tablet_service.cc:1468] Processing CreateTablet for tablet 331364bf121c480483b11ff32993e1ec (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250902 01:57:45.750954 31115 raft_consensus.cc:357] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.708614 31062 tablet_service.cc:1468] Processing CreateTablet for tablet 703afdc575144ab78740c24ef2852ca8 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250902 01:57:45.769969 30913 tablet_service.cc:1468] Processing CreateTablet for tablet 9685199b6c7b42f391ab48fdf86d71ad (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250902 01:57:45.746824 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3ca4f287b94f60b51ef1768df7bb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.781569 31062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703afdc575144ab78740c24ef2852ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.781759 30914 tablet_service.cc:1468] Processing CreateTablet for tablet aeb7c457a3f741b5b4d057f06363c614 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250902 01:57:45.776589 31115 raft_consensus.cc:383] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.787827 31115 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.768891 30916 tablet_service.cc:1468] Processing CreateTablet for tablet ade302bdd69a4fa7876c90a1fbd87b68 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250902 01:57:45.788678 31115 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.788688 31065 tablet_service.cc:1468] Processing CreateTablet for tablet cba8e025cdf94c449fa56870398f3419 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250902 01:57:45.792028 31065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba8e025cdf94c449fa56870398f3419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.793661 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331364bf121c480483b11ff32993e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.793802 31063 tablet_service.cc:1468] Processing CreateTablet for tablet 7f650d4e1b5f47febbda93daccba0e4f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250902 01:57:45.794535 30989 tablet_service.cc:1468] Processing CreateTablet for tablet 63fa3c697d3e40afb9a01578cbe72700 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250902 01:57:45.794873 31063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f650d4e1b5f47febbda93daccba0e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.797173 31064 tablet_service.cc:1468] Processing CreateTablet for tablet 45b3012a0ae843acbfd03ed33966d585 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250902 01:57:45.802104 31056 tablet_service.cc:1468] Processing CreateTablet for tablet d8ce2ebbb1c24a6c998649ed3656707a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250902 01:57:45.803396 31056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ce2ebbb1c24a6c998649ed3656707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.805590 30983 tablet_service.cc:1468] Processing CreateTablet for tablet 38649bfc334c4db6921eb3b12130e10d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250902 01:57:45.806903 31059 tablet_service.cc:1468] Processing CreateTablet for tablet 122e9f7b6c1d4295b7d12445ed0116ed (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250902 01:57:45.810307 30984 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba5aa5dce214071a172ea365440f0dc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250902 01:57:45.811617 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349593d2d3574cd6ab281c105756f435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.816619 31115 ts_tablet_manager.cc:1428] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.069s user 0.004s sys 0.002s
I20250902 01:57:45.817765 31115 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:45.827879 31064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45b3012a0ae843acbfd03ed33966d585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.829944 30986 tablet_service.cc:1468] Processing CreateTablet for tablet ede06f18c6574c57857475151e3767d2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250902 01:57:45.784933 31123 raft_consensus.cc:695] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:45.839179 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38649bfc334c4db6921eb3b12130e10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.840791 30912 tablet_service.cc:1468] Processing CreateTablet for tablet f0f24bac96dc42dba9adc1f6679c0d48 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250902 01:57:45.841410 30909 tablet_service.cc:1468] Processing CreateTablet for tablet 59bc91ec50764c7bb821a4288bea8d16 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250902 01:57:45.842382 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f24bac96dc42dba9adc1f6679c0d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.843400 30916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade302bdd69a4fa7876c90a1fbd87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.844493 30913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9685199b6c7b42f391ab48fdf86d71ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.848366 30915 tablet_service.cc:1468] Processing CreateTablet for tablet de2e42094ddd41f99d12a4972ee07b9f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250902 01:57:45.849606 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2e42094ddd41f99d12a4972ee07b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.852347 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb7c457a3f741b5b4d057f06363c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.854527 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59bc91ec50764c7bb821a4288bea8d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.854584 31123 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:45.858098 31059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122e9f7b6c1d4295b7d12445ed0116ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.865500 30985 tablet_service.cc:1468] Processing CreateTablet for tablet f160859afd5945949c030026bfe3ac61 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250902 01:57:45.869756 31116 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:45.870216 31116 ts_tablet_manager.cc:1397] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.412s user 0.019s sys 0.009s
I20250902 01:57:45.872973 31116 raft_consensus.cc:357] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.873569 31116 raft_consensus.cc:383] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.873845 31116 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.874251 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede06f18c6574c57857475151e3767d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.875801 30989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fa3c697d3e40afb9a01578cbe72700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.876459 30985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f160859afd5945949c030026bfe3ac61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.876992 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba5aa5dce214071a172ea365440f0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.877866 31118 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:45.878330 31118 ts_tablet_manager.cc:1397] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.625s user 0.014s sys 0.016s
I20250902 01:57:45.880784 30908 tablet_service.cc:1468] Processing CreateTablet for tablet 744f11902df74dea8914fdd4a992259d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250902 01:57:45.882287 30908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744f11902df74dea8914fdd4a992259d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.890136 31120 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:45.890564 31120 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.897224 30910 tablet_service.cc:1468] Processing CreateTablet for tablet 416020230983423c9f843070a7893658 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250902 01:57:45.898608 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416020230983423c9f843070a7893658. 1 dirs total, 0 dirs full, 0 dirs failed
W20250902 01:57:45.905398 30811 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 61204.4us
I20250902 01:57:45.910144 31115 tablet_bootstrap.cc:654] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:45.913072 31120 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:45.915555 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:45.916868 30965 leader_election.cc:343] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.918301 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:45.922276 30967 leader_election.cc:343] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.922684 30967 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:45.923874 30907 tablet_service.cc:1468] Processing CreateTablet for tablet 2e3ca4f287b94f60b51ef1768df7bb70 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250902 01:57:45.925102 30907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e3ca4f287b94f60b51ef1768df7bb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.928222 31116 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.944583 31120 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:45.950327 31118 raft_consensus.cc:357] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.951059 31118 raft_consensus.cc:383] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:45.951378 31118 raft_consensus.cc:738] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:45.952154 31118 consensus_queue.cc:260] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.954545 30915 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2dd8f5f814a7d8b48a74020055750 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250902 01:57:45.960976 31132 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:45.971925 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2dd8f5f814a7d8b48a74020055750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.966861 30982 tablet_service.cc:1468] Processing CreateTablet for tablet a2d2c9c23b6145c9b6601f7e78f1f43a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250902 01:57:45.971463 31132 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:45.974113 31132 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:45.970639 30911 tablet_service.cc:1468] Processing CreateTablet for tablet 0692c5029b5e4ab088324d038ff964be (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250902 01:57:45.981691 30910 tablet_service.cc:1468] Processing CreateTablet for tablet b03cb35549a346d28baa21d708d665e2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250902 01:57:45.975638 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:45.983211 30988 tablet_service.cc:1468] Processing CreateTablet for tablet 543054d6241649d59bb40156217a73bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250902 01:57:45.985993 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0692c5029b5e4ab088324d038ff964be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250902 01:57:45.986446 31039 leader_election.cc:343] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:45.986925 30910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03cb35549a346d28baa21d708d665e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:45.988310 30909 tablet_service.cc:1468] Processing CreateTablet for tablet d920df318de242f797c216f655b63218 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250902 01:57:45.989897 30913 tablet_service.cc:1468] Processing CreateTablet for tablet 349593d2d3574cd6ab281c105756f435 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250902 01:57:45.993278 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d920df318de242f797c216f655b63218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.000147 31116 ts_tablet_manager.cc:1428] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.130s user 0.006s sys 0.002s
I20250902 01:57:45.999996 30823 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.003144 30914 tablet_service.cc:1468] Processing CreateTablet for tablet 63fa3c697d3e40afb9a01578cbe72700 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250902 01:57:46.008491 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.008770 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fa3c697d3e40afb9a01578cbe72700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.008709 30987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53742 (request call id 41) took 1028 ms. Trace:
I20250902 01:57:46.009186 30987 rpcz_store.cc:276] 0902 01:57:44.980031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.683342 (+703311us) service_pool.cc:225] Handling call
0902 01:57:45.685115 (+ 1773us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.685204 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:45.851710 (+166506us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:45.985242 (+133532us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:45.997330 (+ 12088us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191882 00007fc607eb4b29 00007fc607017ea8 00007fc6070dc773 00007fc6070d853f 00007fc6070d8cfd 00007fc607089209 00007fc600638bf4 00007fc600638b82
0902 01:57:46.008661 (+ 11331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9,"spinlock_wait_cycles":14613888}
W20250902 01:57:46.012038 31038 leader_election.cc:343] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.012439 31038 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.013128 31123 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.014281 31120 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.014456 30987 tablet_service.cc:1468] Processing CreateTablet for tablet 703afdc575144ab78740c24ef2852ca8 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250902 01:57:46.014719 31120 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.015681 30987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703afdc575144ab78740c24ef2852ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.016606 31120 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.018126 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:46.019675 30965 leader_election.cc:343] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.021976 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:46.023285 30967 leader_election.cc:343] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.023710 30967 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:46.025326 31120 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.028105 30990 tablet_service.cc:1468] Processing CreateTablet for tablet d8ce2ebbb1c24a6c998649ed3656707a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250902 01:57:46.030288 30916 tablet_service.cc:1468] Processing CreateTablet for tablet 38649bfc334c4db6921eb3b12130e10d (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250902 01:57:46.031100 31115 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.031567 31115 ts_tablet_manager.cc:1397] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.214s user 0.021s sys 0.008s
I20250902 01:57:46.033408 30986 tablet_service.cc:1468] Processing CreateTablet for tablet cba8e025cdf94c449fa56870398f3419 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250902 01:57:46.020660 31116 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.034096 31115 raft_consensus.cc:357] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.034722 31115 raft_consensus.cc:383] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.035005 31115 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.035754 31115 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.045159 31115 ts_tablet_manager.cc:1428] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250902 01:57:46.046150 31115 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.049825 30911 tablet_service.cc:1468] Processing CreateTablet for tablet a2d2c9c23b6145c9b6601f7e78f1f43a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250902 01:57:46.053622 31118 ts_tablet_manager.cc:1428] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.175s user 0.006s sys 0.019s
I20250902 01:57:46.054538 30908 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba5aa5dce214071a172ea365440f0dc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250902 01:57:46.058161 30983 tablet_service.cc:1468] Processing CreateTablet for tablet 7f650d4e1b5f47febbda93daccba0e4f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250902 01:57:46.059777 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d2c9c23b6145c9b6601f7e78f1f43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.062181 31118 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.063768 30913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349593d2d3574cd6ab281c105756f435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.063958 30988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543054d6241649d59bb40156217a73bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.067907 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ce2ebbb1c24a6c998649ed3656707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.069025 30986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba8e025cdf94c449fa56870398f3419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.073894 30914 tablet_service.cc:1468] Processing CreateTablet for tablet 543054d6241649d59bb40156217a73bc (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250902 01:57:46.075376 31116 tablet_bootstrap.cc:654] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.075870 30983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f650d4e1b5f47febbda93daccba0e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.045189 30912 tablet_service.cc:1468] Processing CreateTablet for tablet f160859afd5945949c030026bfe3ac61 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250902 01:57:46.085314 30984 tablet_service.cc:1468] Processing CreateTablet for tablet 45b3012a0ae843acbfd03ed33966d585 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250902 01:57:46.086524 30984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45b3012a0ae843acbfd03ed33966d585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.092643 31118 tablet_bootstrap.cc:654] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.099715 30916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38649bfc334c4db6921eb3b12130e10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.100425 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d2c9c23b6145c9b6601f7e78f1f43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.106719 30915 tablet_service.cc:1468] Processing CreateTablet for tablet 703afdc575144ab78740c24ef2852ca8 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250902 01:57:46.107544 30908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba5aa5dce214071a172ea365440f0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.112221 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543054d6241649d59bb40156217a73bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.114526 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f160859afd5945949c030026bfe3ac61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.122304 31115 tablet_bootstrap.cc:654] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.130640 30909 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38794 (request call id 42) took 1001 ms. Trace:
I20250902 01:57:46.131665 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703afdc575144ab78740c24ef2852ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.133384 30990 tablet_service.cc:1468] Processing CreateTablet for tablet 122e9f7b6c1d4295b7d12445ed0116ed (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250902 01:57:46.134790 30991 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53742 (request call id 42) took 1154 ms. Trace:
I20250902 01:57:46.131108 30909 rpcz_store.cc:276] 0902 01:57:45.128942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.987388 (+858446us) service_pool.cc:225] Handling call
0902 01:57:45.990763 (+ 3375us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.990869 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.095863 (+104994us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.130119 (+ 34256us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.130565 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62}
I20250902 01:57:46.137703 30991 rpcz_store.cc:276] 0902 01:57:44.980651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.705172 (+724521us) service_pool.cc:225] Handling call
0902 01:57:45.706918 (+ 1746us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.709474 (+ 2556us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:45.943684 (+234210us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.134332 (+190648us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.134737 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25331}
I20250902 01:57:46.140615 31116 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.140851 30990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122e9f7b6c1d4295b7d12445ed0116ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.141088 31116 ts_tablet_manager.cc:1397] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.140s user 0.015s sys 0.011s
I20250902 01:57:46.143827 31116 raft_consensus.cc:357] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.144486 31116 raft_consensus.cc:383] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.144766 31116 raft_consensus.cc:738] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.145166 30989 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53742 (request call id 45) took 1051 ms. Trace:
I20250902 01:57:46.130724 30913 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38794 (request call id 44) took 1001 ms. Trace:
I20250902 01:57:46.145593 30989 rpcz_store.cc:276] 0902 01:57:45.093640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.793726 (+700086us) service_pool.cc:225] Handling call
0902 01:57:45.795259 (+ 1533us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.795343 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.048553 (+253210us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.144573 (+ 96020us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.145102 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17544}
I20250902 01:57:46.151822 30913 rpcz_store.cc:276] 0902 01:57:45.129467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.989015 (+859548us) service_pool.cc:225] Handling call
0902 01:57:46.006150 (+ 17135us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:46.007578 (+ 1428us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.121953 (+114375us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.130210 (+ 8257us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.130651 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:46.159338 30915 tablet_service.cc:1468] Processing CreateTablet for tablet d8ce2ebbb1c24a6c998649ed3656707a (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250902 01:57:46.167975 30912 tablet_service.cc:1468] Processing CreateTablet for tablet 45b3012a0ae843acbfd03ed33966d585 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250902 01:57:46.163766 30915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ce2ebbb1c24a6c998649ed3656707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.145509 30911 tablet_service.cc:1468] Processing CreateTablet for tablet ede06f18c6574c57857475151e3767d2 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250902 01:57:46.178139 30913 tablet_service.cc:1468] Processing CreateTablet for tablet 7f650d4e1b5f47febbda93daccba0e4f (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250902 01:57:46.178428 30909 tablet_service.cc:1468] Processing CreateTablet for tablet cba8e025cdf94c449fa56870398f3419 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250902 01:57:46.180555 30907 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38794 (request call id 43) took 1051 ms. Trace:
I20250902 01:57:46.180902 30907 rpcz_store.cc:276] 0902 01:57:45.129200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.906767 (+777567us) service_pool.cc:225] Handling call
0902 01:57:45.924674 (+ 17907us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.924758 (+ 84us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.081853 (+157095us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.180063 (+ 98210us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.180509 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:46.164213 31116 consensus_queue.cc:260] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.185972 30982 tablet_service.cc:1468] Processing CreateTablet for tablet 558db87de6fa4361bf0ca91e27d34943 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250902 01:57:46.187506 30982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 558db87de6fa4361bf0ca91e27d34943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.188306 31116 ts_tablet_manager.cc:1428] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.047s user 0.004s sys 0.004s
I20250902 01:57:46.188140 31118 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.189313 31116 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.192756 30912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45b3012a0ae843acbfd03ed33966d585. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.199309 30913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f650d4e1b5f47febbda93daccba0e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.193239 31118 ts_tablet_manager.cc:1397] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.020s
I20250902 01:57:46.201480 31115 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.192989 30907 tablet_service.cc:1468] Processing CreateTablet for tablet 122e9f7b6c1d4295b7d12445ed0116ed (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250902 01:57:46.202090 31115 ts_tablet_manager.cc:1397] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.007s
I20250902 01:57:46.202598 30907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122e9f7b6c1d4295b7d12445ed0116ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.207911 30909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba8e025cdf94c449fa56870398f3419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.208281 31118 raft_consensus.cc:357] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.209877 30911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede06f18c6574c57857475151e3767d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.215095 30910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38794 (request call id 41) took 1086 ms. Trace:
I20250902 01:57:46.215461 30910 rpcz_store.cc:276] 0902 01:57:45.128612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.980694 (+852082us) service_pool.cc:225] Handling call
0902 01:57:45.986467 (+ 5773us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.986548 (+ 81us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.087020 (+100472us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.199330 (+112310us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.215039 (+ 15709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:46.217793 30914 tablet_service.cc:1468] Processing CreateTablet for tablet 558db87de6fa4361bf0ca91e27d34943 (DEFAULT_TABLE table=test-workload [id=7afa88e279034b01a4a86f528bb93af2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250902 01:57:46.218916 30914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 558db87de6fa4361bf0ca91e27d34943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:57:46.214376 31118 raft_consensus.cc:383] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.228327 31118 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.229043 31118 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.230957 31118 ts_tablet_manager.cc:1428] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.031s user 0.001s sys 0.004s
I20250902 01:57:46.233597 31118 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.240541 31118 tablet_bootstrap.cc:654] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.256108 31116 tablet_bootstrap.cc:654] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.258239 31115 raft_consensus.cc:357] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.263952 30916 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38794 (request call id 46) took 1134 ms. Trace:
I20250902 01:57:46.267760 30916 rpcz_store.cc:276] 0902 01:57:45.130008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:46.029059 (+899051us) service_pool.cc:225] Handling call
0902 01:57:46.031298 (+ 2239us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:46.031420 (+ 122us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:46.184833 (+153413us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.232822 (+ 47989us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.263885 (+ 31063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:46.268890 31115 raft_consensus.cc:383] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.268754 31123 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:46.269194 31115 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.269862 31115 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.271325 31118 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.271783 31118 ts_tablet_manager.cc:1397] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.038s user 0.019s sys 0.007s
I20250902 01:57:46.272401 31115 ts_tablet_manager.cc:1428] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.070s user 0.010s sys 0.000s
I20250902 01:57:46.273640 31115 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.274304 31118 raft_consensus.cc:357] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:46.274984 31118 raft_consensus.cc:383] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.275266 31118 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.276429 31118 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:46.278757 31118 ts_tablet_manager.cc:1428] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250902 01:57:46.280198 31118 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.280630 31135 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.280860 31115 tablet_bootstrap.cc:654] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.281082 31135 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.285013 31116 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.285636 31116 ts_tablet_manager.cc:1397] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.007s
I20250902 01:57:46.292508 31135 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.293679 31135 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.294821 31135 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.304308 31135 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.311748 31116 raft_consensus.cc:357] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.312680 31116 raft_consensus.cc:383] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.313005 31116 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.325717 31118 tablet_bootstrap.cc:654] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.329711 31116 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.334465 31151 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.342401 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.344763 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.345506 31000 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
W20250902 01:57:46.347038 30889 leader_election.cc:343] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.347865 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.348910 31135 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.349349 31135 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.335040 31151 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:46.349731 31135 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.353935 31116 ts_tablet_manager.cc:1428] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
W20250902 01:57:46.355919 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:46.356779 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.362005 31151 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.362792 31135 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.365733 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.366271 31000 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.370087 31118 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.371063 31118 ts_tablet_manager.cc:1397] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.006s
I20250902 01:57:46.372967 31000 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.373729 31135 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.373977 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.374596 31118 raft_consensus.cc:357] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.375401 31135 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.375761 31118 raft_consensus.cc:383] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.375890 31135 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:46.376084 31118 raft_consensus.cc:738] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.376675 31135 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.376896 31118 consensus_queue.cc:260] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.386492 31118 ts_tablet_manager.cc:1428] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.015s user 0.004s sys 0.007s
I20250902 01:57:46.387463 31118 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.390736 30926 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.388321 31116 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.398418 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.398945 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.399024 31075 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.399741 31132 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:46.400902 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.401314 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.402858 30822 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.404991 31115 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.405476 31115 ts_tablet_manager.cc:1397] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.132s user 0.056s sys 0.068s
I20250902 01:57:46.407445 31118 tablet_bootstrap.cc:654] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.409878 31115 raft_consensus.cc:357] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.410600 31115 raft_consensus.cc:383] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.409449 31116 tablet_bootstrap.cc:654] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.410924 31115 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.412529 31115 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.415742 30892 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.418251 31115 ts_tablet_manager.cc:1428] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20250902 01:57:46.419260 31115 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.423940 31135 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.424299 31135 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.424602 31135 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.430235 31115 tablet_bootstrap.cc:654] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:46.432130 30892 leader_election.cc:343] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:46.433035 30892 leader_election.cc:343] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.434069 30892 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.431344 31135 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.437546 31118 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.437992 31118 ts_tablet_manager.cc:1397] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.000s
I20250902 01:57:46.438881 31146 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.437980 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.439415 31146 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.442350 31146 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.444468 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.445027 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:46.445842 30967 leader_election.cc:343] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:46.446321 30965 leader_election.cc:343] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.446892 30965 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:46.448364 31146 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.450278 31135 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:46.451411 30889 leader_election.cc:343] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.452365 31151 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.454303 31118 raft_consensus.cc:357] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.455183 31118 raft_consensus.cc:383] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.455615 31118 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.456401 31118 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.457881 31116 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.458262 31116 ts_tablet_manager.cc:1397] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.070s user 0.024s sys 0.021s
I20250902 01:57:46.459242 31118 ts_tablet_manager.cc:1428] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.021s user 0.004s sys 0.001s
I20250902 01:57:46.460125 31116 raft_consensus.cc:357] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.460736 31116 raft_consensus.cc:383] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.460769 31118 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.461143 31116 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.461841 31116 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.463157 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.463704 31074 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.469923 31118 tablet_bootstrap.cc:654] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.470419 31074 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.471496 30892 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.472563 31135 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.472895 31116 ts_tablet_manager.cc:1428] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20250902 01:57:46.472985 31135 raft_consensus.cc:695] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:46.474893 31135 consensus_queue.cc:237] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.476867 31115 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.477448 31115 ts_tablet_manager.cc:1397] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.058s user 0.021s sys 0.029s
I20250902 01:57:46.478726 31116 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.480648 31115 raft_consensus.cc:357] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.484850 31115 raft_consensus.cc:383] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.485126 31115 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.485787 31115 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.488732 31116 tablet_bootstrap.cc:654] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.490348 31115 ts_tablet_manager.cc:1428] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250902 01:57:46.491346 31115 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.501754 31116 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.502421 31116 ts_tablet_manager.cc:1397] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.010s
I20250902 01:57:46.505334 31116 raft_consensus.cc:357] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.506136 31116 raft_consensus.cc:383] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.506465 31116 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.507210 31116 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.508922 31146 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.509509 31115 tablet_bootstrap.cc:654] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.509347 31146 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.510020 31116 ts_tablet_manager.cc:1428] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250902 01:57:46.511106 31116 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.511144 31146 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.512952 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.513352 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:46.514142 30965 leader_election.cc:343] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.514815 31118 tablet_bootstrap.cc:492] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.515277 31118 ts_tablet_manager.cc:1397] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.016s
I20250902 01:57:46.515703 30822 catalog_manager.cc:5582] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.517776 31118 raft_consensus.cc:357] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:46.518249 30967 leader_election.cc:343] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.518435 31118 raft_consensus.cc:383] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.518725 31118 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.519457 31118 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.520179 31116 tablet_bootstrap.cc:654] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.520442 30967 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:46.521585 31146 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.521895 31118 ts_tablet_manager.cc:1428] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:46.522789 31118 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.528358 31118 tablet_bootstrap.cc:654] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.530246 31123 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.530766 31123 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.533295 31123 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:46.535310 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.537369 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.540695 31001 raft_consensus.cc:2391] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:46.541846 31038 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.542973 31123 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.543073 31115 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.543864 31115 ts_tablet_manager.cc:1397] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.019s
I20250902 01:57:46.544708 31116 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.545142 31116 ts_tablet_manager.cc:1397] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.009s
I20250902 01:57:46.546182 31115 raft_consensus.cc:357] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.546779 31115 raft_consensus.cc:383] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.547034 31115 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.547451 31135 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.547583 31116 raft_consensus.cc:357] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.547868 31135 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.547684 31115 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.548350 31116 raft_consensus.cc:383] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.548641 31118 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.548719 31116 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.549108 31118 ts_tablet_manager.cc:1397] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.011s
I20250902 01:57:46.549676 31135 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.549540 31116 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.551741 31118 raft_consensus.cc:357] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.552093 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.552416 31118 raft_consensus.cc:383] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.552727 31118 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.553161 31115 ts_tablet_manager.cc:1428] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20250902 01:57:46.553236 30892 leader_election.cc:343] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.554203 31115 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.554071 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.554661 31001 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.554428 31118 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.555718 30889 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:46.556723 31118 ts_tablet_manager.cc:1428] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250902 01:57:46.556998 31157 raft_consensus.cc:2802] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.557417 31157 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.557675 31118 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.557806 31157 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.562374 31116 ts_tablet_manager.cc:1428] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250902 01:57:46.564018 31118 tablet_bootstrap.cc:654] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.568264 31123 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:46.568354 31116 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.570684 31115 tablet_bootstrap.cc:654] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.575059 31157 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.576817 31116 tablet_bootstrap.cc:654] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.578266 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.578788 31001 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.579363 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.579638 31118 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.580165 31118 ts_tablet_manager.cc:1397] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.013s
W20250902 01:57:46.580484 30892 leader_election.cc:343] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.580976 31157 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.584463 31001 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.583772 31118 raft_consensus.cc:357] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.585108 31118 raft_consensus.cc:383] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.585461 31118 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.585635 30889 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:46.586731 31157 raft_consensus.cc:2802] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.586925 31115 tablet_bootstrap.cc:492] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.586364 31118 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.587320 31115 ts_tablet_manager.cc:1397] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.004s
I20250902 01:57:46.588176 31157 raft_consensus.cc:695] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:46.589020 31157 consensus_queue.cc:237] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.589493 31118 ts_tablet_manager.cc:1428] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250902 01:57:46.589771 31115 raft_consensus.cc:357] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.590396 31115 raft_consensus.cc:383] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.590649 31115 raft_consensus.cc:738] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.590662 31118 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.591435 31115 consensus_queue.cc:260] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.597710 31115 ts_tablet_manager.cc:1428] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250902 01:57:46.598278 31118 tablet_bootstrap.cc:654] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.598584 31115 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.604118 31115 tablet_bootstrap.cc:654] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.609270 31116 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.609830 31116 ts_tablet_manager.cc:1397] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.025s
I20250902 01:57:46.612975 31116 raft_consensus.cc:357] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.613790 31116 raft_consensus.cc:383] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.614075 31116 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.614746 31116 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.617197 31116 ts_tablet_manager.cc:1428] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250902 01:57:46.618484 31116 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.619443 30822 catalog_manager.cc:5582] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.625260 31123 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.625943 31116 tablet_bootstrap.cc:654] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.625690 31123 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.628777 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.629357 31001 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:46.630470 31038 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.631093 31132 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.631423 31132 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.631835 31132 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.636343 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.637094 30925 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:46.636351 31123 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:46.638494 31132 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.644479 31115 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.645026 31115 ts_tablet_manager.cc:1397] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.047s user 0.014s sys 0.016s
I20250902 01:57:46.646446 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:46.647178 30925 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.649201 31115 raft_consensus.cc:357] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.649801 31115 raft_consensus.cc:383] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.649868 31118 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.650125 31115 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.650519 31118 ts_tablet_manager.cc:1397] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.028s
I20250902 01:57:46.650789 31115 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.652951 31118 raft_consensus.cc:357] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.653566 31118 raft_consensus.cc:383] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.653803 31118 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.654492 31118 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.655916 31115 ts_tablet_manager.cc:1428] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250902 01:57:46.656787 31115 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.659742 31118 ts_tablet_manager.cc:1428] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250902 01:57:46.660595 31118 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.662595 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.663334 31001 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.665648 31118 tablet_bootstrap.cc:654] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.674121 31115 tablet_bootstrap.cc:654] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.677729 31118 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.678267 31118 ts_tablet_manager.cc:1397] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20250902 01:57:46.681322 31118 raft_consensus.cc:357] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.682185 31118 raft_consensus.cc:383] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.682493 31118 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.683455 31118 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.686113 31001 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:46.688234 31038 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.690085 31123 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.691264 31123 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:46.692111 31123 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.686661 31132 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:46.695951 31118 ts_tablet_manager.cc:1428] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20250902 01:57:46.700507 30925 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:46.702277 31118 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.708895 31116 tablet_bootstrap.cc:492] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.709235 31116 ts_tablet_manager.cc:1397] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.091s user 0.039s sys 0.027s
I20250902 01:57:46.711922 31118 tablet_bootstrap.cc:654] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.711225 31116 raft_consensus.cc:357] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.712563 31116 raft_consensus.cc:383] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.706856 30821 catalog_manager.cc:5582] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:46.712846 31116 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.713569 31116 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.714839 31115 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.714821 31146 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.715559 31115 ts_tablet_manager.cc:1397] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.059s user 0.028s sys 0.018s
I20250902 01:57:46.715878 31116 ts_tablet_manager.cc:1428] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250902 01:57:46.715605 31146 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.716780 31116 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.718163 31115 raft_consensus.cc:357] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.718818 31115 raft_consensus.cc:383] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.719192 31115 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.719880 31115 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.721143 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.722106 31115 ts_tablet_manager.cc:1428] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:46.721724 31074 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:46.723093 31116 tablet_bootstrap.cc:654] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.723115 31115 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.723224 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.723935 30925 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:46.725194 30967 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.725973 31120 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.726306 31120 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.726598 31120 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.730760 31115 tablet_bootstrap.cc:654] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.742401 31157 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.742826 31157 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.744714 31157 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.746297 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.746245 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.747241 31074 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
W20250902 01:57:46.748400 30889 leader_election.cc:343] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.750437 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.751394 31157 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.751749 31157 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.752081 31157 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.752105 31146 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.753623 31120 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.757545 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:46.758090 30925 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.758597 31157 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.761451 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:46.762656 30889 leader_election.cc:343] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.763480 31157 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.764580 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.765229 31074 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.777742 30925 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:46.780056 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.780812 31075 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.788336 31157 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:46.789112 31075 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:46.790381 30967 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.791198 31146 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.796228 31120 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.798314 31074 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.800421 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:46.801383 31157 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.801812 31157 raft_consensus.cc:695] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:46.802742 31157 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.803246 31146 raft_consensus.cc:695] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:46.804186 31146 consensus_queue.cc:237] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.806648 31135 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.807078 31135 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.808995 31135 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.810361 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.810945 31001 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.812353 30889 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.813007 31135 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.813301 31135 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.813586 31135 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.813791 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.814359 31074 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.816823 31115 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.817269 31115 ts_tablet_manager.cc:1397] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.094s user 0.035s sys 0.051s
I20250902 01:57:46.819604 31115 raft_consensus.cc:357] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.821602 31116 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.822732 31115 raft_consensus.cc:383] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.822991 31116 ts_tablet_manager.cc:1397] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.106s user 0.042s sys 0.052s
I20250902 01:57:46.823076 31115 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.823810 31115 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.825804 31115 ts_tablet_manager.cc:1428] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250902 01:57:46.826602 31115 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.826274 31116 raft_consensus.cc:357] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.819710 31135 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.828047 31135 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.821295 31118 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.829942 31118 ts_tablet_manager.cc:1397] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.128s user 0.021s sys 0.027s
I20250902 01:57:46.828914 30821 catalog_manager.cc:5582] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.830825 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.832206 31118 raft_consensus.cc:357] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.832624 31115 tablet_bootstrap.cc:654] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.832796 31118 raft_consensus.cc:383] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.833058 31118 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.829396 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:46.833851 31074 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.833892 31118 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.836290 31118 ts_tablet_manager.cc:1428] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250902 01:57:46.837275 31118 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.831360 31001 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.843147 31116 raft_consensus.cc:383] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.843497 31116 raft_consensus.cc:738] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.844420 31116 consensus_queue.cc:260] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.846338 31001 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.847360 31116 ts_tablet_manager.cc:1428] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20250902 01:57:46.847435 30889 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.849014 31116 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.854281 31118 tablet_bootstrap.cc:654] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.859351 31135 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:46.859786 31135 raft_consensus.cc:695] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:46.860492 31135 consensus_queue.cc:237] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.861117 31116 tablet_bootstrap.cc:654] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.863355 31074 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:46.866837 31192 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.868101 31192 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.918061 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.918874 31000 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.857543 31191 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.062s
I20250902 01:57:46.921269 30889 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.921507 31191 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.918121 31192 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.922695 31192 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.923316 31192 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.923715 31192 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.934998 31116 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:46.935446 31116 ts_tablet_manager.cc:1397] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.008s
I20250902 01:57:46.935561 31118 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:46.936115 31118 ts_tablet_manager.cc:1397] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.026s
I20250902 01:57:46.938035 31116 raft_consensus.cc:357] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.938845 31116 raft_consensus.cc:383] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.940456 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.941128 31179 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.941286 31115 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:46.941723 31115 ts_tablet_manager.cc:1397] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.115s user 0.031s sys 0.056s
I20250902 01:57:46.941596 31179 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.939146 31116 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.946338 31179 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:46.946717 31074 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.947825 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.947885 31115 raft_consensus.cc:357] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.948441 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.948694 31115 raft_consensus.cc:383] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.948966 31001 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:46.922106 31191 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.943964 31116 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.950145 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.951026 31123 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.951303 31001 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.951380 31123 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.949123 31115 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.952044 31123 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.952615 31115 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.954763 31115 ts_tablet_manager.cc:1428] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250902 01:57:46.956491 31191 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.939640 31118 raft_consensus.cc:357] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.958634 31118 raft_consensus.cc:383] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:46.959013 31118 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:46.959729 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:46.960039 31123 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.960309 30999 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:46.961318 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:46.962589 31118 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.965790 30823 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:46.967304 31146 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.967818 31146 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.968459 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.969686 31146 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.970551 31116 ts_tablet_manager.cc:1428] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250902 01:57:46.971907 31115 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:46.972468 31146 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:46.973843 31135 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:46.973744 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:46.974047 30892 leader_election.cc:343] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.974306 31135 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:46.974628 31135 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.973029 31146 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.975284 31191 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:46.976557 31191 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:46.978560 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:46.979419 31073 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:46.971489 31116 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:46.980142 31146 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:46.981948 31123 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:46.983434 30965 leader_election.cc:343] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:46.984318 31192 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:46.984582 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:46.985428 30925 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:46.987002 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:46.988265 30965 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:46.988672 31118 ts_tablet_manager.cc:1428] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20250902 01:57:46.989369 31146 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.989530 31118 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:46.993614 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:46.994318 30999 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:46.995316 31116 tablet_bootstrap.cc:654] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:46.994670 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.000576 31135 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.001196 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.003372 30999 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.004364 31115 tablet_bootstrap.cc:654] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.005314 31146 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.005911 31192 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:47.009074 30967 leader_election.cc:343] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.010007 30967 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:47.010192 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.011467 31073 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:47.011595 31185 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.016274 31184 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.016379 30892 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.016780 31184 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.018373 31118 tablet_bootstrap.cc:654] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.018615 31184 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.020212 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:47.018630 31191 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.021209 31123 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.022456 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.022498 31123 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.022652 31073 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:47.023865 30965 leader_election.cc:343] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.024716 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.024729 31123 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.009984 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.029058 31135 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:47.041587 30967 leader_election.cc:343] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.042088 30967 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:47.042960 30999 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.056061 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.048732 31185 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250902 01:57:47.061147 30892 leader_election.cc:343] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.081317 30999 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:47.083165 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.084280 31135 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.084758 31135 raft_consensus.cc:695] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:47.085521 31135 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.089700 30822 catalog_manager.cc:5582] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:47.096928 31192 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.098275 31115 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.098629 31115 ts_tablet_manager.cc:1397] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.001s
I20250902 01:57:47.101227 31118 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.101536 31118 ts_tablet_manager.cc:1397] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.112s user 0.014s sys 0.027s
I20250902 01:57:47.103931 31115 raft_consensus.cc:357] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.104568 31115 raft_consensus.cc:383] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.104821 31115 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.105588 31116 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.105392 31115 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.105868 31116 ts_tablet_manager.cc:1397] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.135s user 0.016s sys 0.006s
I20250902 01:57:47.107084 31115 ts_tablet_manager.cc:1428] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20250902 01:57:47.108966 31179 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.108428 31116 raft_consensus.cc:357] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.110162 31116 raft_consensus.cc:383] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.110376 31116 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.110839 31116 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.109408 31179 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.111440 31115 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.112812 31116 ts_tablet_manager.cc:1428] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250902 01:57:47.115680 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.116676 31179 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.116560 30926 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.115474 31116 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.119089 31039 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:47.120188 31179 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.120611 31179 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.121028 31179 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.121382 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.122160 31135 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250902 01:57:47.122483 31038 leader_election.cc:343] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.123919 31118 raft_consensus.cc:357] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.124387 31118 raft_consensus.cc:383] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.124643 31118 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.125236 31118 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.130646 31179 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.131567 31116 tablet_bootstrap.cc:654] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.121001 31115 tablet_bootstrap.cc:654] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.127504 31118 ts_tablet_manager.cc:1428] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250902 01:57:47.133397 31118 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.146003 30823 catalog_manager.cc:5582] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:47.153199 31132 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.155562 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.156164 30926 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.163707 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:47.165202 31038 leader_election.cc:343] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.166115 31118 tablet_bootstrap.cc:654] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.166148 31179 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.172361 31115 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.173739 31115 ts_tablet_manager.cc:1397] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.064s user 0.023s sys 0.012s
I20250902 01:57:47.173780 30926 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.175474 31039 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.176723 31115 raft_consensus.cc:357] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.177678 31132 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.178325 31115 raft_consensus.cc:383] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.179013 31115 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.178426 31132 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.181382 31132 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.182765 31123 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.188755 30822 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:47.180730 31115 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.205576 31115 ts_tablet_manager.cc:1428] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20250902 01:57:47.206514 31115 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.206857 31135 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.207274 31135 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.210261 31135 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.211035 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:47.212707 30889 leader_election.cc:343] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.215363 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.216567 31118 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.217000 31118 ts_tablet_manager.cc:1397] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.084s user 0.016s sys 0.006s
W20250902 01:57:47.217334 30892 leader_election.cc:343] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.218299 30892 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.219220 31116 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.219597 31116 ts_tablet_manager.cc:1397] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.104s user 0.019s sys 0.017s
I20250902 01:57:47.219638 31192 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.222412 31116 raft_consensus.cc:357] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.223075 31116 raft_consensus.cc:383] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.223328 31116 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.223759 31118 raft_consensus.cc:357] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.223951 31116 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.224691 31118 raft_consensus.cc:383] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.224997 31118 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.224396 31115 tablet_bootstrap.cc:654] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.226234 31116 ts_tablet_manager.cc:1428] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:47.227212 31116 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.227644 31118 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.228022 30822 catalog_manager.cc:5582] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:47.230677 31118 ts_tablet_manager.cc:1428] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250902 01:57:47.233108 31116 tablet_bootstrap.cc:654] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.236914 31118 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.242738 31118 tablet_bootstrap.cc:654] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.248025 31115 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.248376 31115 ts_tablet_manager.cc:1397] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.016s
I20250902 01:57:47.250458 31116 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.250936 31116 ts_tablet_manager.cc:1397] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.009s
I20250902 01:57:47.250838 31115 raft_consensus.cc:357] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.251523 31115 raft_consensus.cc:383] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.251832 31115 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.252568 31115 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.254170 31116 raft_consensus.cc:357] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.254784 31116 raft_consensus.cc:383] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.255055 31116 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.255756 31116 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.257779 31116 ts_tablet_manager.cc:1428] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20250902 01:57:47.258186 31115 ts_tablet_manager.cc:1428] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250902 01:57:47.259061 31115 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.260246 31116 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.264432 31179 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.265290 31115 tablet_bootstrap.cc:654] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.264989 31179 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.266217 31116 tablet_bootstrap.cc:654] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.269300 31179 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.269896 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.270416 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.271494 31118 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.271977 31118 ts_tablet_manager.cc:1397] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.011s
I20250902 01:57:47.272121 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.272357 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.272697 30925 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.273245 31179 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.273590 31179 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.273902 31179 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.278504 31192 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.281642 31215 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.282321 31215 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.286478 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.286947 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.287770 31001 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.288431 31179 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.289220 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.290244 31132 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.290643 31132 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.291014 31132 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.296221 31215 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.303794 31132 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:47.306176 31039 leader_election.cc:343] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.307772 31118 raft_consensus.cc:357] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.308246 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.308478 31118 raft_consensus.cc:383] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.308733 31118 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.308787 31001 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.309171 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.309491 31118 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.309837 30925 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.305938 31179 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.318197 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.319204 31118 ts_tablet_manager.cc:1428] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250902 01:57:47.318507 31192 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.327464 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.328043 30926 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.329142 30925 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.330442 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.332793 31116 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.333212 31116 ts_tablet_manager.cc:1397] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.020s
I20250902 01:57:47.334421 30926 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.335378 31039 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:47.326843 31179 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.336964 31123 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.336807 31179 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.335418 31116 raft_consensus.cc:357] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.339637 31116 raft_consensus.cc:383] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.339954 31116 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.341019 31179 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.336138 31215 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.341717 31215 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.341864 31116 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.342490 31215 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.344388 31116 ts_tablet_manager.cc:1428] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250902 01:57:47.345299 31116 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.345651 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.345778 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.345309 31115 tablet_bootstrap.cc:492] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.346405 31000 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:47.346853 31115 ts_tablet_manager.cc:1397] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.015s
I20250902 01:57:47.347649 31118 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.352396 31039 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.354154 31132 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.357513 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.358295 31000 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.365783 31132 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.367672 31151 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.369748 31000 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.371033 31115 raft_consensus.cc:357] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.374025 31185 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.376688 31115 raft_consensus.cc:383] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.378715 31115 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.379357 31115 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.381716 31115 ts_tablet_manager.cc:1428] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20250902 01:57:47.382834 31115 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.388590 31132 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.385264 31123 raft_consensus.cc:695] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.393321 31123 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:47.400288 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:47.406903 31115 tablet_bootstrap.cc:654] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.411795 31116 tablet_bootstrap.cc:654] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.415028 30821 catalog_manager.cc:5582] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:47.417079 31146 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.416824 30821 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:47.423962 31115 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.423877 31146 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.425836 31146 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.427919 31146 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.428994 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.429411 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.429785 30925 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:47.430034 31073 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:47.431157 30967 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.424454 31115 ts_tablet_manager.cc:1397] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.000s
I20250902 01:57:47.435041 31115 raft_consensus.cc:357] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.435700 31115 raft_consensus.cc:383] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.435943 31115 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.436528 31115 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.444235 31118 tablet_bootstrap.cc:654] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.445534 31184 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.455308 31184 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.455672 31184 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.461475 31184 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.447259 31115 ts_tablet_manager.cc:1428] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250902 01:57:47.447657 31135 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.465595 31118 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.466092 31118 ts_tablet_manager.cc:1397] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.011s
I20250902 01:57:47.467833 31184 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.468441 31118 raft_consensus.cc:357] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.468894 31118 raft_consensus.cc:383] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.469142 31118 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.470422 31118 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.471460 31116 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.471912 31116 ts_tablet_manager.cc:1397] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.127s user 0.019s sys 0.037s
I20250902 01:57:47.472240 31118 ts_tablet_manager.cc:1428] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:47.473081 31118 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.474716 31116 raft_consensus.cc:357] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.475332 31116 raft_consensus.cc:383] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.475577 31116 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.475972 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.476473 31073 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.476217 31116 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.477917 31116 ts_tablet_manager.cc:1428] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:47.478227 31118 tablet_bootstrap.cc:654] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.480090 31151 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.480631 31115 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.481916 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.482437 30926 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.488729 31116 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.479262 31157 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.491624 31157 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.493590 31157 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.495261 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:47.496752 30892 leader_election.cc:343] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.497440 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.483493 31073 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:47.498045 31001 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:47.498621 30967 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.499131 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.499404 31184 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.500003 31184 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:47.500782 31184 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.500087 31157 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.503831 31157 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.504195 31157 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.514674 31157 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.521322 31115 tablet_bootstrap.cc:654] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.523638 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.524197 31001 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.514578 30926 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:47.535331 30822 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:47.537261 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.537112 31157 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:47.539196 30892 leader_election.cc:343] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.542291 31116 tablet_bootstrap.cc:654] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.548483 31123 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.548483 31001 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:47.552065 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.554363 31135 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.554944 31135 raft_consensus.cc:695] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:47.555248 31118 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.555646 31118 ts_tablet_manager.cc:1397] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.007s
I20250902 01:57:47.555871 31135 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.557650 31118 raft_consensus.cc:357] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.558211 31118 raft_consensus.cc:383] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.558457 31118 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.559044 31118 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.561127 31118 ts_tablet_manager.cc:1428] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20250902 01:57:47.561875 31118 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.567263 31118 tablet_bootstrap.cc:654] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.574110 31192 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.576787 31115 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.577198 31115 ts_tablet_manager.cc:1397] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.011s
I20250902 01:57:47.585023 31157 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.585567 31157 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.587733 31157 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.589565 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.589723 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:47.591183 30889 leader_election.cc:343] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:47.591804 30892 leader_election.cc:343] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.592339 30892 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.593291 31157 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.593351 31115 raft_consensus.cc:357] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.595180 31115 raft_consensus.cc:383] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.595963 31115 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.597280 31115 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.600706 31115 ts_tablet_manager.cc:1428] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250902 01:57:47.603350 31115 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.607882 31118 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.608340 31118 ts_tablet_manager.cc:1397] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.015s
I20250902 01:57:47.609496 31116 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.609858 31115 tablet_bootstrap.cc:654] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.609970 31116 ts_tablet_manager.cc:1397] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.131s user 0.022s sys 0.019s
I20250902 01:57:47.610805 31118 raft_consensus.cc:357] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.610559 30823 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:47.611465 31118 raft_consensus.cc:383] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.611765 31118 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.613543 31146 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.613782 31118 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.613999 31146 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.615816 31146 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.616106 31118 ts_tablet_manager.cc:1428] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250902 01:57:47.616990 31118 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.617658 31116 raft_consensus.cc:357] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.618628 31116 raft_consensus.cc:383] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.619004 31116 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.619961 31116 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.622694 31118 tablet_bootstrap.cc:654] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.623960 31116 ts_tablet_manager.cc:1428] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250902 01:57:47.624399 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.625329 31116 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.626590 31123 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.629076 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.629544 31118 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.629719 30924 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:47.629940 31118 ts_tablet_manager.cc:1397] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20250902 01:57:47.631172 30965 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:47.632205 31123 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.632225 31118 raft_consensus.cc:357] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.632623 31116 tablet_bootstrap.cc:654] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.632943 31118 raft_consensus.cc:383] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.633232 31118 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.634469 31118 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.636144 31146 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.637059 31118 ts_tablet_manager.cc:1428] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250902 01:57:47.637972 31118 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.639840 31115 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.640390 31115 ts_tablet_manager.cc:1397] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.010s
I20250902 01:57:47.643409 31115 raft_consensus.cc:357] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.644204 31115 raft_consensus.cc:383] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.644517 31115 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.645452 31146 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.645253 31115 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.648301 31115 ts_tablet_manager.cc:1428] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20250902 01:57:47.649117 31115 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.650662 31116 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.651518 31116 ts_tablet_manager.cc:1397] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.010s
I20250902 01:57:47.653921 31116 raft_consensus.cc:357] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.660946 31116 raft_consensus.cc:383] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.661237 31116 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.661813 31116 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.656351 31118 tablet_bootstrap.cc:654] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.664419 31116 ts_tablet_manager.cc:1428] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250902 01:57:47.665132 31116 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.668242 31225 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.656514 31132 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.657874 31115 tablet_bootstrap.cc:654] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.670332 31116 tablet_bootstrap.cc:654] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.670362 31132 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.677525 31132 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.679854 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.680804 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:47.681797 31038 leader_election.cc:343] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:47.682524 31039 leader_election.cc:343] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.683715 31135 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.684329 31135 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.683090 31039 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:47.688730 31135 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.688920 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.689718 31001 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:47.692019 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:47.693676 30892 leader_election.cc:343] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.694981 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.696604 31179 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:47.699741 31135 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.700222 31135 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.700629 31135 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.737419 31146 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.738018 31146 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.741901 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.742645 31074 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:47.743724 31146 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.745406 30967 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.745556 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.746423 31146 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.749575 31146 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.750078 31146 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:47.747056 30965 leader_election.cc:343] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.764902 31146 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.766767 31146 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.768066 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.778988 31074 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.779553 31132 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.773590 31115 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:47.784070 31115 ts_tablet_manager.cc:1397] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.032s
I20250902 01:57:47.786856 31115 raft_consensus.cc:357] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.773923 31157 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.787742 31157 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.789494 31157 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.790549 31115 raft_consensus.cc:383] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.790930 31115 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.791724 31115 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.777400 31135 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.796034 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.796582 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.794377 31132 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.797327 31000 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:47.797686 31132 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.789499 31074 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:47.800145 31115 ts_tablet_manager.cc:1428] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250902 01:57:47.800107 30967 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.800843 31146 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
W20250902 01:57:47.797250 30892 leader_election.cc:343] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.801621 31146 raft_consensus.cc:695] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:47.802778 31132 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.802428 31146 consensus_queue.cc:237] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.802408 30889 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.801297 31118 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.803869 31118 ts_tablet_manager.cc:1397] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.166s user 0.059s sys 0.067s
I20250902 01:57:47.803941 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.804642 31000 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.806244 31038 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.807794 31123 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.808861 31115 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:47.809249 31157 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.816155 31179 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.808072 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:47.827708 31157 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.828231 31179 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.828485 31157 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:47.829270 30892 leader_election.cc:343] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.830308 31118 raft_consensus.cc:357] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.830139 31179 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.830947 31118 raft_consensus.cc:383] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.831247 31118 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.831848 31118 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.832075 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.832111 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.832660 30924 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.833123 31000 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.833473 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.833824 31118 ts_tablet_manager.cc:1428] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.030s user 0.006s sys 0.010s
I20250902 01:57:47.834759 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:47.837481 31118 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.836129 31179 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.825002 31123 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.809356 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.810644 31135 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:47.844969 31039 leader_election.cc:343] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.840205 30998 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:47.848526 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:47.849866 31132 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:47.849761 31157 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.850358 31132 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:47.850802 31132 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.851581 31157 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.852887 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:47.854887 30892 leader_election.cc:343] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.861016 31132 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.841585 31123 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.867259 31116 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:47.867678 31116 ts_tablet_manager.cc:1397] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.203s user 0.035s sys 0.108s
I20250902 01:57:47.869359 31118 tablet_bootstrap.cc:654] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.869994 31116 raft_consensus.cc:357] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.870956 31116 raft_consensus.cc:383] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.871251 31116 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.872236 31116 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.873275 31123 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.873934 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.874464 31116 ts_tablet_manager.cc:1428] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250902 01:57:47.874894 31123 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.875409 31116 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:47.874413 30999 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.877413 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:47.879053 30965 leader_election.cc:343] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.879050 30997 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.871250 31000 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:47.876618 31132 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:47.885293 30822 catalog_manager.cc:5582] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:47.886152 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:47.888020 30996 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:47.892027 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.896822 30999 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.897881 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.898590 31132 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.899000 31132 raft_consensus.cc:695] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.899928 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:47.899729 31132 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.899821 31179 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.900457 30926 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:47.909116 31039 leader_election.cc:343] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.912715 31184 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:47.899439 30996 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:47.915634 30926 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.915685 31184 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.917407 31123 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:47.918545 31184 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:47.913848 30997 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:47.919719 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:47.920388 31115 tablet_bootstrap.cc:654] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.918954 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:47.925468 31038 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:47.928372 31179 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:47.928776 31179 raft_consensus.cc:695] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:47.929446 31179 consensus_queue.cc:237] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.931325 30822 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
W20250902 01:57:47.932368 30967 leader_election.cc:343] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:47.936444 31116 tablet_bootstrap.cc:654] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:47.938805 31118 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:47.939239 31118 ts_tablet_manager.cc:1397] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.020s
I20250902 01:57:47.941895 31118 raft_consensus.cc:357] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.942843 31118 raft_consensus.cc:383] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:47.943316 31118 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:47.944296 31118 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.946554 31118 ts_tablet_manager.cc:1428] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:47.947777 31118 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:47.957089 31184 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:47.971038 31157 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
W20250902 01:57:47.971246 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 52375.7us
I20250902 01:57:47.971727 31157 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:47.972625 31157 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:47.978520 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:47.979292 30926 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:47.995433 31075 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.017493 31118 tablet_bootstrap.cc:654] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.019850 31116 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.020285 31116 ts_tablet_manager.cc:1397] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.145s user 0.030s sys 0.017s
I20250902 01:57:48.024544 31116 raft_consensus.cc:357] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.024240 30824 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.025434 31116 raft_consensus.cc:383] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.025818 31116 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.026515 31116 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.010843 30889 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.030582 31157 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.031036 31157 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:48.031409 31116 ts_tablet_manager.cc:1428] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20250902 01:57:48.031808 31157 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.032534 31116 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.045748 30965 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.046159 31115 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.046509 31115 ts_tablet_manager.cc:1397] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.243s user 0.027s sys 0.037s
I20250902 01:57:48.048628 31185 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.048717 31146 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250902 01:57:48.055627 31185 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.056862 31118 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.057053 31185 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.057317 31118 ts_tablet_manager.cc:1397] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.011s
I20250902 01:57:48.060010 31118 raft_consensus.cc:357] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.063271 31118 raft_consensus.cc:383] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.063630 31118 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.063308 31115 raft_consensus.cc:357] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.064038 31115 raft_consensus.cc:383] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.064296 31115 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.064231 31118 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.064970 31115 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.065996 31118 ts_tablet_manager.cc:1428] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250902 01:57:48.066546 31118 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.068892 31115 ts_tablet_manager.cc:1428] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250902 01:57:48.069707 31115 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.069522 31185 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.077227 31116 tablet_bootstrap.cc:654] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.083762 31135 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.080941 30821 catalog_manager.cc:5582] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:48.085556 31185 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.085249 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:48.086403 30924 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.091563 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:48.092128 31074 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.099854 30820 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.109692 30924 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.114988 30965 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:48.115320 31118 tablet_bootstrap.cc:654] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.115741 31185 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.116348 31115 tablet_bootstrap.cc:654] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.116763 31185 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:48.117802 31185 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.124073 31157 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.124637 31157 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.127488 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.127743 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:48.128815 30892 leader_election.cc:343] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:48.129352 30889 leader_election.cc:343] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.126967 31157 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.130280 31116 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.129769 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.130682 31116 ts_tablet_manager.cc:1397] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.005s
I20250902 01:57:48.131500 31135 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.113193 31074 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.133862 31116 raft_consensus.cc:357] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.134394 31116 raft_consensus.cc:383] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.134616 31116 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.135123 31116 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.141690 31116 ts_tablet_manager.cc:1428] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250902 01:57:48.143092 31116 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.144788 31118 tablet_bootstrap.cc:492] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.145212 31118 ts_tablet_manager.cc:1397] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.007s
I20250902 01:57:48.146310 31157 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.147374 31118 raft_consensus.cc:357] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.148010 31118 raft_consensus.cc:383] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.148159 31116 tablet_bootstrap.cc:654] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.148326 31118 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.151113 31115 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.151515 31115 ts_tablet_manager.cc:1397] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.002s
I20250902 01:57:48.151141 31118 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.155606 31115 raft_consensus.cc:357] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.156193 31115 raft_consensus.cc:383] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.156349 31118 ts_tablet_manager.cc:1428] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250902 01:57:48.156455 31115 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.157737 31118 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.157598 31115 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.159404 31115 ts_tablet_manager.cc:1428] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250902 01:57:48.163997 31118 tablet_bootstrap.cc:654] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.169013 31118 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.169384 31118 ts_tablet_manager.cc:1397] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20250902 01:57:48.171335 31116 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.171458 31118 raft_consensus.cc:357] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.171787 31116 ts_tablet_manager.cc:1397] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20250902 01:57:48.172010 31118 raft_consensus.cc:383] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.172293 31118 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.172858 31118 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.173230 30821 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.173969 31116 raft_consensus.cc:357] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.174568 31116 raft_consensus.cc:383] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.174798 31116 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.175500 31118 ts_tablet_manager.cc:1428] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250902 01:57:48.175304 31116 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.176594 31118 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.177500 31116 ts_tablet_manager.cc:1428] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:57:48.178300 31116 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.183812 31118 tablet_bootstrap.cc:654] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.160168 31115 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.194590 31115 tablet_bootstrap.cc:654] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.195988 31116 tablet_bootstrap.cc:654] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.199574 31118 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.199975 31118 ts_tablet_manager.cc:1397] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.014s
I20250902 01:57:48.202122 31118 raft_consensus.cc:357] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.202538 31118 raft_consensus.cc:383] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.202735 31118 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.203637 31115 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.204087 31115 ts_tablet_manager.cc:1397] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.003s
I20250902 01:57:48.203570 31118 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.205653 31115 raft_consensus.cc:357] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.206089 31115 raft_consensus.cc:383] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.206279 31115 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.206743 31115 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.208539 31115 ts_tablet_manager.cc:1428] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250902 01:57:48.209306 31115 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.213303 31118 ts_tablet_manager.cc:1428] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250902 01:57:48.213629 31116 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.214033 31116 ts_tablet_manager.cc:1397] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.006s
I20250902 01:57:48.214136 31118 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.217403 31116 raft_consensus.cc:357] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.217964 31116 raft_consensus.cc:383] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.218194 31116 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.219863 31118 tablet_bootstrap.cc:654] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.218813 31116 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.222383 31116 ts_tablet_manager.cc:1428] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20250902 01:57:48.223404 31116 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.225924 31115 tablet_bootstrap.cc:654] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.227319 31157 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.227752 31157 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.228832 31118 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.229156 31118 ts_tablet_manager.cc:1397] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250902 01:57:48.229422 31157 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.230612 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.231505 31118 raft_consensus.cc:357] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.232249 31118 raft_consensus.cc:383] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.232295 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.232563 31118 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.232969 31074 raft_consensus.cc:2391] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:48.233275 31116 tablet_bootstrap.cc:654] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.233294 31118 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.234375 30889 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.235481 31118 ts_tablet_manager.cc:1428] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20250902 01:57:48.237151 31135 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.237311 31157 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.237686 31157 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.240528 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.241061 31074 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:48.241493 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.241645 31157 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.242064 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:48.242432 31135 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.242635 31118 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.243058 31135 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.243458 31135 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.243808 31135 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:48.243851 30889 leader_election.cc:343] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.247625 31118 tablet_bootstrap.cc:654] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.249195 31135 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.252432 31118 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.252890 31118 ts_tablet_manager.cc:1397] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250902 01:57:48.254542 31115 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.254865 31118 raft_consensus.cc:357] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.255270 31115 ts_tablet_manager.cc:1397] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.007s
I20250902 01:57:48.255570 31118 raft_consensus.cc:383] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.255865 31118 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.256659 31118 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.258363 31116 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.258747 31116 ts_tablet_manager.cc:1397] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.002s
I20250902 01:57:48.260278 31115 raft_consensus.cc:357] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.260811 31115 raft_consensus.cc:383] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.261040 31115 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.260941 31116 raft_consensus.cc:357] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.261704 31116 raft_consensus.cc:383] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.261976 31116 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.261953 31115 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.264050 31115 ts_tablet_manager.cc:1428] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250902 01:57:48.264494 31116 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.266476 31135 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.267010 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:48.267930 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:48.268081 30889 leader_election.cc:343] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.268426 31074 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.269507 31118 ts_tablet_manager.cc:1428] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250902 01:57:48.273039 31118 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.274636 31074 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:48.275269 31116 ts_tablet_manager.cc:1428] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20250902 01:57:48.275591 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.276502 31157 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.276877 31157 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.277772 31118 tablet_bootstrap.cc:654] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.278498 31116 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.278455 31157 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.278999 31157 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.280138 31157 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:48.280179 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:48.281364 30892 leader_election.cc:343] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.280973 31157 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.282984 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:48.284891 30889 leader_election.cc:343] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.285249 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.286134 31135 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.287122 31115 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.293888 31115 tablet_bootstrap.cc:654] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.294238 31118 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.294736 31118 ts_tablet_manager.cc:1397] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.006s
I20250902 01:57:48.298594 31116 tablet_bootstrap.cc:654] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.299175 31115 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.299566 31115 ts_tablet_manager.cc:1397] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20250902 01:57:48.297263 31118 raft_consensus.cc:357] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.300304 31118 raft_consensus.cc:383] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.300617 31118 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.301086 31185 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.301532 31185 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.301651 31118 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.303529 30824 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.304744 31118 ts_tablet_manager.cc:1428] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20250902 01:57:48.305168 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.305991 31118 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.306739 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.307312 31074 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:48.308382 30967 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:48.309113 31146 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.309443 31146 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.309748 31146 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.314344 31118 tablet_bootstrap.cc:654] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:48.314766 30965 leader_election.cc:343] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.303467 31185 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.324995 31116 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.325356 31116 ts_tablet_manager.cc:1397] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.047s user 0.017s sys 0.015s
I20250902 01:57:48.325610 31146 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.326274 31118 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.326676 31118 ts_tablet_manager.cc:1397] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.002s
I20250902 01:57:48.328078 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:48.327903 31116 raft_consensus.cc:357] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.328625 31116 raft_consensus.cc:383] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.328876 31116 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
W20250902 01:57:48.329250 30965 leader_election.cc:343] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.329026 31118 raft_consensus.cc:357] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.329648 31118 raft_consensus.cc:383] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.329897 31118 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.329452 31116 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.330534 31118 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.331993 31118 ts_tablet_manager.cc:1428] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20250902 01:57:48.332046 31116 ts_tablet_manager.cc:1428] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250902 01:57:48.332890 31179 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.333165 31116 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.333218 31118 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.333273 31179 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.334993 31179 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:48.336995 31115 raft_consensus.cc:357] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.337600 31115 raft_consensus.cc:383] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.337893 31115 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.338399 31115 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.339213 31146 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.340097 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.341199 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:48.341851 31074 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:48.342512 31038 leader_election.cc:343] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.349423 31074 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.351166 31115 ts_tablet_manager.cc:1428] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250902 01:57:48.364498 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.366353 30924 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:48.366701 31115 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.366566 30925 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.367574 31039 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.370209 31118 tablet_bootstrap.cc:654] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.368263 31185 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250902 01:57:48.372438 30967 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:48.373140 31146 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.373764 31146 raft_consensus.cc:695] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:48.374574 31146 consensus_queue.cc:237] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.375684 31184 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.376106 31184 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.376631 31179 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.377326 31179 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.377624 31179 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.378178 31184 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.379508 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.380074 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:48.380918 30967 leader_election.cc:343] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.380919 30926 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:48.382119 30965 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.383100 31184 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.383438 31184 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.383737 31184 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.383832 31132 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.382961 31123 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.390717 31179 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.392156 31184 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.395309 31135 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.395875 31116 tablet_bootstrap.cc:654] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.395750 31135 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.397689 31135 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.400564 31115 tablet_bootstrap.cc:654] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.401300 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:48.402971 30892 leader_election.cc:343] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.404889 31179 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:48.405030 31215 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.410231 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:48.410615 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.413379 31184 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:48.413892 30889 leader_election.cc:343] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:48.415849 30892 consensus_peers.cc:489] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Couldn't send request to peer 088424002a894fd5bcc11a06af111445. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:48.416364 31038 leader_election.cc:343] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.414184 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.417336 31157 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.418408 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:48.418939 30926 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.419304 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:48.420459 30967 leader_election.cc:343] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.420914 31135 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.441224 31115 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.441712 31115 ts_tablet_manager.cc:1397] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.004s
I20250902 01:57:48.438205 30926 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.447928 30965 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.449440 31146 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.449878 31146 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:48.451805 31215 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:48.453639 31185 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.457932 30824 catalog_manager.cc:5582] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.460217 31146 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.450865 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:48.469398 30925 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.471189 31135 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.471657 31135 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.473345 31135 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.474512 31135 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.474651 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.477101 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:48.482920 30892 leader_election.cc:343] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.490724 31179 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.491194 31179 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.492914 31179 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:48.493227 31184 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.493605 31184 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.494778 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.496295 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.496951 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.498159 31038 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.499323 31191 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.499708 31191 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.500247 31116 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.500612 31116 ts_tablet_manager.cc:1397] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.168s user 0.005s sys 0.010s
I20250902 01:57:48.495457 31184 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.496966 31115 raft_consensus.cc:357] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.502602 31115 raft_consensus.cc:383] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.503629 31132 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.503952 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.505025 31191 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.505151 31116 raft_consensus.cc:357] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.505870 31116 raft_consensus.cc:383] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.506145 31116 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
W20250902 01:57:48.506759 30967 leader_election.cc:343] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.503343 31115 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.507428 31116 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.503608 31118 tablet_bootstrap.cc:492] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.509074 31118 ts_tablet_manager.cc:1397] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.176s user 0.013s sys 0.038s
I20250902 01:57:48.508312 31115 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.524462 31116 ts_tablet_manager.cc:1428] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
W20250902 01:57:48.531199 30965 leader_election.cc:343] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.533187 30965 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:48.534451 31185 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250902 01:57:48.531459 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:48.516750 31001 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:48.532234 31116 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.532485 31115 ts_tablet_manager.cc:1428] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.090s user 0.006s sys 0.002s
I20250902 01:57:48.532660 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.537218 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.537370 31075 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:48.537825 31115 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.537778 30998 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:48.538586 30892 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:48.539368 31191 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.539734 31191 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.540025 31191 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.515436 30925 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:48.557199 31191 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.555078 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.558863 31191 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.559187 31135 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.559417 31135 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.559646 31135 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.562712 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:48.563407 31074 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.564381 31118 raft_consensus.cc:357] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.567749 31118 raft_consensus.cc:383] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.568045 31118 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.568516 31118 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.565608 31135 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.572571 31185 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.574582 31185 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.576416 31118 ts_tablet_manager.cc:1428] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20250902 01:57:48.577484 31185 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.577446 31118 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.578482 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.579018 31075 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:48.580196 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:48.580376 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:48.580894 30996 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.581503 31185 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:48.581882 31185 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:48.582229 31185 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.584023 31039 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.585425 31132 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.585857 31132 raft_consensus.cc:695] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:48.591099 31132 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.590677 30822 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.601114 31115 tablet_bootstrap.cc:654] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.609648 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.610208 30926 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:48.611948 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:48.612669 31074 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:48.613692 31185 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.616999 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:48.617497 30997 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.618204 31185 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:48.622526 30892 leader_election.cc:343] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:48.623436 30892 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:48.624840 31191 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.625407 31191 raft_consensus.cc:695] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:48.626394 31191 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.601572 31135 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.628314 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:48.629042 31072 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.631460 30996 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
W20250902 01:57:48.670949 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 56549.4us
I20250902 01:57:48.694473 31116 tablet_bootstrap.cc:654] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.705070 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:48.705687 30925 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.705745 31074 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.713443 31135 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.711310 30997 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:48.715222 31115 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.715842 31115 ts_tablet_manager.cc:1397] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.178s user 0.030s sys 0.000s
I20250902 01:57:48.720603 30925 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.726756 31118 tablet_bootstrap.cc:654] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.729838 31135 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.731092 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:48.728081 31184 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250902 01:57:48.733845 31146 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.733723 30822 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:48.737140 31135 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.737510 31135 raft_consensus.cc:695] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:48.738093 31135 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.740602 30965 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:48.743652 31185 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:48.748642 31185 raft_consensus.cc:695] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:48.749466 31185 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.749886 31115 raft_consensus.cc:357] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.750792 31115 raft_consensus.cc:383] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.751523 31115 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.752591 31115 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.755784 31072 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:48.761622 30824 catalog_manager.cc:5582] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.772505 31115 ts_tablet_manager.cc:1428] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20250902 01:57:48.773643 31115 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.785382 31307 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.789415 31307 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.803594 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.842548 31132 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:48.843093 31132 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.865298 31132 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:48.867722 31115 tablet_bootstrap.cc:654] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.890974 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:48.892084 30923 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:48.914563 31116 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:48.915022 31116 ts_tablet_manager.cc:1397] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.383s user 0.021s sys 0.006s
I20250902 01:57:48.917856 31116 raft_consensus.cc:357] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.918498 31116 raft_consensus.cc:383] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.918709 31116 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.919476 30923 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.919765 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:48.920493 31072 raft_consensus.cc:2391] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:48.921492 30892 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.924005 30996 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.929453 31185 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.929502 30824 catalog_manager.cc:5582] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:48.934080 31307 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:48.942199 31118 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:48.942625 31118 ts_tablet_manager.cc:1397] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.366s user 0.021s sys 0.057s
I20250902 01:57:48.919273 31116 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.943248 31146 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.944896 31116 ts_tablet_manager.cc:1428] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250902 01:57:48.945700 31116 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:48.945839 31135 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.948727 31184 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.951061 31115 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:48.953567 31115 ts_tablet_manager.cc:1397] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.180s user 0.005s sys 0.013s
I20250902 01:57:48.955727 31115 raft_consensus.cc:357] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.957628 31115 raft_consensus.cc:383] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.957944 31115 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.958469 31116 tablet_bootstrap.cc:654] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.952816 31118 raft_consensus.cc:357] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.958951 31118 raft_consensus.cc:383] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:48.959247 31118 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:48.959803 31118 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.961011 31135 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:48.961983 31118 ts_tablet_manager.cc:1428] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250902 01:57:48.963423 31118 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:48.964823 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:48.958561 31115 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:48.966848 31115 ts_tablet_manager.cc:1428] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250902 01:57:48.967329 31073 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:48.967849 31115 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:48.974192 31118 tablet_bootstrap.cc:654] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:48.981463 31038 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:48.982353 31303 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:48.993608 31314 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:48.997730 30824 catalog_manager.cc:5582] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:49.002583 31116 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.003278 31116 ts_tablet_manager.cc:1397] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.000s
I20250902 01:57:49.006564 31116 raft_consensus.cc:357] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.007175 31116 raft_consensus.cc:383] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.007529 31116 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.009496 31115 tablet_bootstrap.cc:654] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.013631 31116 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.019363 31116 ts_tablet_manager.cc:1428] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250902 01:57:49.020455 31116 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.033823 31132 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.034449 31132 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.036784 31132 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:49.038836 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.045953 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.053754 31118 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.054196 31118 ts_tablet_manager.cc:1397] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.006s
I20250902 01:57:49.054450 31038 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.055119 31116 tablet_bootstrap.cc:654] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.055454 31132 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.058555 31115 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.059026 31115 ts_tablet_manager.cc:1397] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.092s user 0.018s sys 0.005s
I20250902 01:57:49.061384 31115 raft_consensus.cc:357] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.061555 31118 raft_consensus.cc:357] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.062055 31115 raft_consensus.cc:383] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.062146 31118 raft_consensus.cc:383] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.062374 31115 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.062431 31118 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.063162 31115 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.064283 31116 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.064682 31116 ts_tablet_manager.cc:1397] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20250902 01:57:49.065217 31115 ts_tablet_manager.cc:1428] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250902 01:57:49.063118 31118 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.066902 31116 raft_consensus.cc:357] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.067428 31116 raft_consensus.cc:383] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.067664 31116 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.068181 31116 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.068887 31118 ts_tablet_manager.cc:1428] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250902 01:57:49.069550 31118 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.070099 31116 ts_tablet_manager.cc:1428] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:57:49.071156 31116 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.074913 31118 tablet_bootstrap.cc:654] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.076201 31115 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.077056 31116 tablet_bootstrap.cc:654] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.082370 31115 tablet_bootstrap.cc:654] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.083593 30996 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.100067 31116 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.100605 31116 ts_tablet_manager.cc:1397] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.000s
I20250902 01:57:49.106724 31307 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.107313 31307 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.108675 31116 raft_consensus.cc:357] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.109462 31116 raft_consensus.cc:383] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.109519 31307 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.110105 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.110983 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.109735 31116 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
W20250902 01:57:49.112675 30889 leader_election.cc:343] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:49.113229 30892 leader_election.cc:343] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.113108 31116 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.113819 31115 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.114240 31115 ts_tablet_manager.cc:1397] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.017s
I20250902 01:57:49.115093 31116 ts_tablet_manager.cc:1428] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250902 01:57:49.115880 31116 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.115684 31132 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.113621 30892 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.116441 31115 raft_consensus.cc:357] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.117113 31115 raft_consensus.cc:383] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.117385 31115 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.117456 31307 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.118031 31115 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.120934 31115 ts_tablet_manager.cc:1428] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250902 01:57:49.121774 31115 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.122364 31116 tablet_bootstrap.cc:654] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.127411 31184 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.127949 31184 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.129901 31184 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.133368 31118 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.132831 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.133932 31118 ts_tablet_manager.cc:1397] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.065s user 0.022s sys 0.014s
I20250902 01:57:49.134066 31073 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:49.135200 30967 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:49.135931 31184 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.136183 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.136260 31184 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.136744 31184 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.136773 30923 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:49.136878 31118 raft_consensus.cc:357] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.137547 31118 raft_consensus.cc:383] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.137816 31118 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.138645 31118 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.139959 31115 tablet_bootstrap.cc:654] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.141258 31118 ts_tablet_manager.cc:1428] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:49.144910 31307 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.145445 31118 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.152801 31118 tablet_bootstrap.cc:654] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.173704 31184 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:49.174475 30892 consensus_peers.cc:489] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Couldn't send request to peer 088424002a894fd5bcc11a06af111445. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:49.175799 31132 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.177588 31157 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.185278 31116 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.185777 31116 ts_tablet_manager.cc:1397] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.004s
I20250902 01:57:49.207108 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:49.207827 30923 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.209661 31314 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.210224 31314 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.214685 31307 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.223948 30923 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:49.225035 30965 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:49.225941 31185 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:49.226547 31185 raft_consensus.cc:695] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:49.226495 31184 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.227767 31185 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.231048 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:49.231817 31074 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.236697 31115 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.237143 31115 ts_tablet_manager.cc:1397] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.116s user 0.020s sys 0.032s
I20250902 01:57:49.239781 31115 raft_consensus.cc:357] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.240337 31115 raft_consensus.cc:383] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.240584 31115 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.241138 31115 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.243088 31115 ts_tablet_manager.cc:1428] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:49.244261 31116 raft_consensus.cc:357] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.244882 31116 raft_consensus.cc:383] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.245121 31116 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.245618 31116 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.247750 31116 ts_tablet_manager.cc:1428] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.062s user 0.009s sys 0.006s
I20250902 01:57:49.248508 31116 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.249132 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.249716 31115 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
W20250902 01:57:49.250341 31038 leader_election.cc:343] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.256914 31319 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.257390 31319 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.261226 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.261812 31075 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:49.262954 30967 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:49.256501 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.264120 30923 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:49.264897 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.264570 31146 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.265537 31146 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.265619 31314 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:49.264480 31319 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.265878 31146 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:49.267210 30965 leader_election.cc:343] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.278923 31115 tablet_bootstrap.cc:654] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.280807 31319 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.284190 31319 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:49.280483 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.288978 31215 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.291671 31215 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.292016 31215 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.294652 31146 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.298827 31146 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.306946 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:49.313305 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:49.314275 31118 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.314677 31118 ts_tablet_manager.cc:1397] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.169s user 0.011s sys 0.019s
I20250902 01:57:49.300163 31074 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:49.315080 31072 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.325045 31157 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:57:49.313547 30965 leader_election.cc:343] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.350320 31157 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.353009 31115 tablet_bootstrap.cc:492] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.353688 31215 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.354702 31116 tablet_bootstrap.cc:654] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.314497 30824 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:49.356359 31115 ts_tablet_manager.cc:1397] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.017s
I20250902 01:57:49.360903 31192 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.363411 31192 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.374572 31118 raft_consensus.cc:357] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.375041 31118 raft_consensus.cc:383] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.375233 31118 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.376425 31118 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.383272 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.386603 31072 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:49.387019 31192 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.387645 30967 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:49.389915 31074 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.386467 31115 raft_consensus.cc:357] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.392717 31118 ts_tablet_manager.cc:1428] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.078s user 0.002s sys 0.005s
I20250902 01:57:49.395689 31115 raft_consensus.cc:383] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.396082 31115 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.394882 31146 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:49.392935 31157 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.398764 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:49.400148 30889 leader_election.cc:343] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.400990 31146 raft_consensus.cc:695] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:49.401173 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.401808 31075 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:49.396844 31115 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.401824 31146 consensus_queue.cc:237] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.403489 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.402338 31118 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.406770 30997 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:49.407162 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:49.408025 31115 ts_tablet_manager.cc:1428] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.051s user 0.003s sys 0.004s
I20250902 01:57:49.410120 31115 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.408906 31215 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:49.415439 30892 leader_election.cc:343] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.415908 30892 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.416074 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:49.416774 30923 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.416806 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:49.413429 31157 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.418407 31157 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.418736 31157 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:49.419592 31038 leader_election.cc:343] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.420737 31192 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.440716 31116 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.441109 31116 ts_tablet_manager.cc:1397] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.193s user 0.018s sys 0.027s
I20250902 01:57:49.460826 31115 tablet_bootstrap.cc:654] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.461043 31157 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.462847 31319 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.464056 31319 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.464927 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:49.465564 30996 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.466202 31319 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.466877 30923 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:49.469388 31118 tablet_bootstrap.cc:654] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.467655 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.475219 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:49.478102 30965 leader_election.cc:343] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.478281 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.478981 31215 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:49.471526 31157 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.479804 31116 raft_consensus.cc:357] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.480634 31116 raft_consensus.cc:383] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.481892 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:49.482527 31072 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:49.483798 30967 leader_election.cc:343] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.480767 31215 raft_consensus.cc:695] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:49.484275 30967 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:49.485424 31146 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.486150 31215 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.487052 30996 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:49.480947 31116 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.493925 31116 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.498147 31115 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.498533 31115 ts_tablet_manager.cc:1397] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.089s user 0.016s sys 0.002s
I20250902 01:57:49.497335 31072 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:49.499048 30820 catalog_manager.cc:5582] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:49.500231 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:49.500861 31157 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:49.500802 31115 raft_consensus.cc:357] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.501457 31157 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:49.501592 31115 raft_consensus.cc:383] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.501912 31115 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.502420 31115 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.502280 31157 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.504184 31115 ts_tablet_manager.cc:1428] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250902 01:57:49.505044 31115 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.510524 31115 tablet_bootstrap.cc:654] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.513468 30824 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:49.518520 31118 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.519104 31118 ts_tablet_manager.cc:1397] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.117s user 0.007s sys 0.003s
I20250902 01:57:49.522327 31118 raft_consensus.cc:357] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.523200 31118 raft_consensus.cc:383] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.526394 31116 ts_tablet_manager.cc:1428] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.085s user 0.005s sys 0.001s
I20250902 01:57:49.527276 31116 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.532392 31116 tablet_bootstrap.cc:654] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.523561 31118 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.545691 31118 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.561363 31118 ts_tablet_manager.cc:1428] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.042s user 0.003s sys 0.005s
I20250902 01:57:49.561941 31072 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.562527 31118 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.568663 31118 tablet_bootstrap.cc:654] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.570668 31115 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.571097 31115 ts_tablet_manager.cc:1397] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.036s
I20250902 01:57:49.572606 31116 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.573048 31116 ts_tablet_manager.cc:1397] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.046s user 0.015s sys 0.001s
I20250902 01:57:49.575006 31115 raft_consensus.cc:357] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.575868 31115 raft_consensus.cc:383] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.576203 31115 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.576625 31116 raft_consensus.cc:357] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.576947 31115 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.578899 31118 tablet_bootstrap.cc:492] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.584000 31118 ts_tablet_manager.cc:1397] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20250902 01:57:49.585126 31115 ts_tablet_manager.cc:1428] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20250902 01:57:49.585866 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:49.586217 31116 raft_consensus.cc:383] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.586153 31118 raft_consensus.cc:357] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.586839 31118 raft_consensus.cc:383] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.587116 31118 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.586541 31116 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.587752 31118 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.587940 31116 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.589745 31118 ts_tablet_manager.cc:1428] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:57:49.589815 31116 ts_tablet_manager.cc:1428] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250902 01:57:49.579881 30824 catalog_manager.cc:5582] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:49.590593 31116 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.590593 31118 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.585939 31115 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.597373 31115 tablet_bootstrap.cc:654] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.603753 31118 tablet_bootstrap.cc:654] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.609623 31116 tablet_bootstrap.cc:654] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.615466 31118 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.615967 31118 ts_tablet_manager.cc:1397] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20250902 01:57:49.617352 31184 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.617792 31184 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.618268 31118 raft_consensus.cc:357] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.619114 31118 raft_consensus.cc:383] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.619392 31118 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.619648 31184 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.620070 31118 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.621279 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.622563 31118 ts_tablet_manager.cc:1428] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20250902 01:57:49.623461 30967 leader_election.cc:343] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.623862 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.624146 31118 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.624497 30923 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:49.626653 31116 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.627882 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:49.629004 31184 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.630549 31116 ts_tablet_manager.cc:1397] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.002s
I20250902 01:57:49.631711 31184 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.632211 31184 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.634985 31118 tablet_bootstrap.cc:654] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.646368 31118 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.646824 31118 ts_tablet_manager.cc:1397] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.000s
I20250902 01:57:49.647140 31116 raft_consensus.cc:357] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.648044 31116 raft_consensus.cc:383] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.648384 31116 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.649425 31116 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.650352 31118 raft_consensus.cc:357] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.651060 31118 raft_consensus.cc:383] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.651376 31118 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.652099 31118 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.652886 31116 ts_tablet_manager.cc:1428] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20250902 01:57:49.654073 31116 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.654681 31118 ts_tablet_manager.cc:1428] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:49.655615 31118 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.657861 31184 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.660254 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:49.660770 30923 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.661805 31116 tablet_bootstrap.cc:654] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.665084 31118 tablet_bootstrap.cc:654] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.667876 30923 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:49.669162 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:49.670209 31319 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:49.670766 31319 raft_consensus.cc:695] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:49.671831 31319 consensus_queue.cc:237] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.673862 31116 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.674011 31118 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.674335 31116 ts_tablet_manager.cc:1397] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20250902 01:57:49.674546 31118 ts_tablet_manager.cc:1397] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20250902 01:57:49.677109 31116 raft_consensus.cc:357] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.677896 31116 raft_consensus.cc:383] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.677567 31118 raft_consensus.cc:357] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.678200 31116 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.678431 31118 raft_consensus.cc:383] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.678829 31118 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.679553 31118 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.685179 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:49.685626 31118 ts_tablet_manager.cc:1428] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20250902 01:57:49.686452 30967 leader_election.cc:343] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.686753 31118 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.694540 31116 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.696117 31118 tablet_bootstrap.cc:654] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.703986 31116 ts_tablet_manager.cc:1428] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250902 01:57:49.705055 31116 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.708331 31118 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.708877 31118 ts_tablet_manager.cc:1397] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20250902 01:57:49.713675 31116 tablet_bootstrap.cc:654] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.714131 31184 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.711903 31118 raft_consensus.cc:357] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.714942 31118 raft_consensus.cc:383] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.715286 31118 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.716171 31118 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.717308 31115 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.717875 31115 ts_tablet_manager.cc:1397] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.132s user 0.055s sys 0.057s
I20250902 01:57:49.718390 31118 ts_tablet_manager.cc:1428] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20250902 01:57:49.719414 31118 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.725744 31118 tablet_bootstrap.cc:654] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.725904 31115 raft_consensus.cc:357] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.726786 31115 raft_consensus.cc:383] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.727128 31115 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.727887 31115 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.731376 31343 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.731325 31115 ts_tablet_manager.cc:1428] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250902 01:57:49.732043 31343 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.732868 31115 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.733837 31116 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.734228 31116 ts_tablet_manager.cc:1397] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.013s
I20250902 01:57:49.735661 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.736519 31116 raft_consensus.cc:357] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:49.736896 30892 leader_election.cc:343] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.737182 31118 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.737766 31118 ts_tablet_manager.cc:1397] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20250902 01:57:49.737208 31116 raft_consensus.cc:383] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.738202 31116 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.738929 31116 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.739336 31115 tablet_bootstrap.cc:654] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.740512 31118 raft_consensus.cc:357] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.740801 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.741305 31118 raft_consensus.cc:383] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.741634 31118 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
W20250902 01:57:49.742085 30889 leader_election.cc:343] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.742522 30889 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.743269 31343 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.743319 31157 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.743937 31116 ts_tablet_manager.cc:1428] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250902 01:57:49.742285 31118 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.745138 31116 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.745378 31118 ts_tablet_manager.cc:1428] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250902 01:57:49.746153 31118 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.746034 31184 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.751932 31116 tablet_bootstrap.cc:654] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.757059 31118 tablet_bootstrap.cc:654] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.760675 31115 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.761101 31115 ts_tablet_manager.cc:1397] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20250902 01:57:49.763330 31115 raft_consensus.cc:357] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.763994 31115 raft_consensus.cc:383] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.764228 31115 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.764797 31115 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.766810 31115 ts_tablet_manager.cc:1428] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:57:49.767604 31115 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.773694 31115 tablet_bootstrap.cc:654] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.776206 31118 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.776661 31118 ts_tablet_manager.cc:1397] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.001s
I20250902 01:57:49.780733 31116 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.781422 31116 ts_tablet_manager.cc:1397] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.011s
I20250902 01:57:49.788057 31115 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:49.788450 31115 ts_tablet_manager.cc:1397] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.002s
I20250902 01:57:49.788985 31184 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.789410 31184 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.790556 31115 raft_consensus.cc:357] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.791144 31184 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.787211 31116 raft_consensus.cc:357] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.791473 31115 raft_consensus.cc:383] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.791707 31116 raft_consensus.cc:383] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.791811 31115 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.792212 31116 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.792497 31115 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.792927 31116 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.796279 31116 ts_tablet_manager.cc:1428] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250902 01:57:49.797132 31116 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.783720 31118 raft_consensus.cc:357] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.792965 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.801803 30924 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:49.800066 31118 raft_consensus.cc:383] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.803738 31118 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.804388 31118 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.806354 31118 ts_tablet_manager.cc:1428] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.029s user 0.004s sys 0.000s
I20250902 01:57:49.806416 31185 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.807258 31118 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:49.811393 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.814853 30967 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:49.809266 30822 catalog_manager.cc:5582] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:49.816051 31185 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.819324 31116 tablet_bootstrap.cc:654] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.820190 31157 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.820581 31157 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.822512 31157 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.824916 31115 ts_tablet_manager.cc:1428] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250902 01:57:49.825729 31115 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:49.825672 31157 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.826355 31157 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.828912 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.832655 31118 tablet_bootstrap.cc:654] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.833349 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.836656 31157 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:49.832463 30892 leader_election.cc:343] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:49.840392 30889 leader_election.cc:343] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.840721 30889 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.841766 31157 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.860929 31184 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:49.862023 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.863013 31072 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:49.877226 30924 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.877713 31184 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.878222 31184 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.879818 31115 tablet_bootstrap.cc:654] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:49.881456 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.869926 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:49.883049 31184 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:49.887794 30967 leader_election.cc:343] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:49.887362 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.887074 30889 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:49.891558 30925 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
W20250902 01:57:49.932883 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45846.6us
I20250902 01:57:49.938628 31343 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.938786 30965 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:49.956112 31345 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.956601 31118 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:49.956552 31345 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.957139 31118 ts_tablet_manager.cc:1397] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.026s
I20250902 01:57:49.958621 31146 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:49.952548 31184 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.959486 31146 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:49.959867 31146 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:49.959641 31184 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.961462 31184 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.961920 31184 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.961760 31118 raft_consensus.cc:357] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.962634 31118 raft_consensus.cc:383] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.962950 31118 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.965404 31116 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:49.965535 31157 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:49.965797 31116 ts_tablet_manager.cc:1397] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.169s user 0.016s sys 0.011s
I20250902 01:57:49.965214 31118 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.970533 31116 raft_consensus.cc:357] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.971238 31345 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:49.971493 31116 raft_consensus.cc:383] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:49.975103 31116 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:49.976225 31116 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.978258 31116 ts_tablet_manager.cc:1428] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250902 01:57:49.983004 31146 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.985340 31146 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:49.983111 31215 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:49.986152 31215 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:49.987774 31215 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:49.989493 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.990774 31185 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:49.990207 30924 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:49.979387 31116 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:49.992728 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:49.993345 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:49.997852 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:49.998623 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.003680 31319 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.006549 31319 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.007278 31307 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.007814 31307 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.008275 31319 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.012450 31307 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.013043 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.013474 30922 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:50.014876 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.015497 31073 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:50.018432 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.018951 30999 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:50.019156 30892 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:57:50.015772 30967 leader_election.cc:343] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.020107 31157 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.020418 31157 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.020612 31157 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.031859 31118 ts_tablet_manager.cc:1428] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.074s user 0.012s sys 0.026s
I20250902 01:57:50.032933 31118 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:50.020315 31039 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.033840 31377 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250902 01:57:50.038703 30965 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:57:50.039456 31038 leader_election.cc:343] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.040773 31115 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.041155 31115 ts_tablet_manager.cc:1397] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.216s user 0.015s sys 0.010s
I20250902 01:57:50.044648 31215 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.045799 31215 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:57:50.045995 30967 leader_election.cc:343] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.046082 31215 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.042361 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.051898 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.052343 30922 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.055039 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.055636 30923 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:50.055727 31157 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.058475 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:50.058957 31073 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.060241 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.061333 30965 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:50.062124 31378 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.062443 31378 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.062800 31378 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.063870 31157 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.065160 31307 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.065660 31307 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.068661 31319 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.069022 31319 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.069335 31319 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.079180 31115 raft_consensus.cc:357] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.081246 31115 raft_consensus.cc:383] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.081465 31115 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.083087 31115 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.090958 30922 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
W20250902 01:57:50.079694 31039 leader_election.cc:343] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.093303 31039 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.094182 30965 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.094261 31376 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.097971 31073 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:50.099512 30892 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.099951 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.100697 31074 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:50.101783 30892 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.102936 31380 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.103096 31116 tablet_bootstrap.cc:654] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.103313 31307 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.096624 31146 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.105727 31185 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.109248 31146 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:50.100708 31343 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.110119 31343 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:50.110809 31343 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.106436 31380 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.113803 31380 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.118238 31115 ts_tablet_manager.cc:1428] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.077s user 0.005s sys 0.004s
I20250902 01:57:50.121268 31115 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.124286 31184 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.124706 31184 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.128068 31184 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.132004 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.133301 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:50.134307 30967 consensus_peers.cc:489] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 -> Peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Couldn't send request to peer 088424002a894fd5bcc11a06af111445. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:50.134872 30967 leader_election.cc:343] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.136409 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.136516 31215 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.136982 31001 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:50.137816 31072 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:50.138597 31319 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.139034 31215 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.118595 31146 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.140751 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:50.140931 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.141487 30923 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:50.143039 31319 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.143209 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:50.144513 30967 leader_election.cc:343] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.153158 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.130784 31307 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.154157 30925 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.155952 31307 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.155136 31157 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.158027 31157 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.158330 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.158890 30924 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.160660 31157 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:50.162519 30889 leader_election.cc:343] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.165968 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.166972 30925 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
W20250902 01:57:50.167367 30892 leader_election.cc:343] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.168776 31377 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.169175 31377 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.170892 31377 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.149281 30965 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:57:50.173694 30889 leader_election.cc:343] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.175251 31157 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.176734 31378 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.178646 31115 tablet_bootstrap.cc:654] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.179132 31378 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.180074 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.180347 31001 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:50.181294 31307 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.183983 30924 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:50.185336 31118 tablet_bootstrap.cc:654] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:50.197479 31038 leader_election.cc:343] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.200634 30820 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.185199 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.173583 31380 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.204794 31380 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.204720 31377 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250902 01:57:50.205982 30892 leader_election.cc:343] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.189927 31116 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.191356 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:50.206842 31116 ts_tablet_manager.cc:1397] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.228s user 0.018s sys 0.000s
I20250902 01:57:50.206867 31073 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:50.208066 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.209198 31116 raft_consensus.cc:357] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.209785 31116 raft_consensus.cc:383] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.210062 31116 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.210201 31039 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.210927 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:50.211061 31116 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.211413 31074 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.211431 31377 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.212054 31377 raft_consensus.cc:695] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:50.213063 31116 ts_tablet_manager.cc:1428] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:50.212808 31377 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.213769 31116 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.215269 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.215867 31001 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:50.175122 31319 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.216676 31319 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.216987 31319 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.220750 31074 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:50.221307 30965 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.221746 30892 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.222574 31380 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.222939 31378 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.223011 31380 raft_consensus.cc:695] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:50.223851 31378 raft_consensus.cc:695] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:50.224045 31380 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.224707 31378 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.230612 31116 tablet_bootstrap.cc:654] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.234382 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.235495 30822 catalog_manager.cc:5582] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:50.241866 31115 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.242303 31115 ts_tablet_manager.cc:1397] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.016s
I20250902 01:57:50.225310 31319 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.245230 31319 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.246155 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:50.247231 30967 leader_election.cc:343] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.247746 31118 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:50.248262 31118 ts_tablet_manager.cc:1397] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.216s user 0.006s sys 0.007s
I20250902 01:57:50.252282 31115 raft_consensus.cc:357] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.252461 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.253211 30924 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:50.254699 31118 raft_consensus.cc:357] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.255332 31118 raft_consensus.cc:383] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.255630 31118 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.257678 31115 raft_consensus.cc:383] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.257952 31115 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.258492 31115 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.259867 30822 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.260581 31115 ts_tablet_manager.cc:1428] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250902 01:57:50.261415 31115 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.263799 31118 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.264703 30820 catalog_manager.cc:5582] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.265962 31118 ts_tablet_manager.cc:1428] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.017s user 0.002s sys 0.004s
I20250902 01:57:50.266753 31118 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:50.272830 31116 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.273293 31116 ts_tablet_manager.cc:1397] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.000s
I20250902 01:57:50.276482 31116 raft_consensus.cc:357] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.277011 31116 raft_consensus.cc:383] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.277254 31116 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.277560 31118 tablet_bootstrap.cc:654] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.277971 31116 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.279927 31115 tablet_bootstrap.cc:654] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.280733 31116 ts_tablet_manager.cc:1428] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:50.282845 31116 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.286365 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.286877 30926 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.289711 31377 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.290120 31377 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.291953 31377 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.293809 30926 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:50.295737 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:50.296263 31001 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.298213 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:50.299938 31038 leader_election.cc:343] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.302197 31001 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
W20250902 01:57:50.304754 31039 leader_election.cc:343] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.305122 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.306028 31377 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.307582 30965 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:50.308526 31378 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250902 01:57:50.310447 30889 leader_election.cc:343] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.310565 31116 tablet_bootstrap.cc:654] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.310952 30889 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.316578 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.317709 31157 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.318035 31157 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.318307 31157 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.315344 30820 catalog_manager.cc:5582] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.321919 31343 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.322412 31343 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.315100 31380 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.323230 31115 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.323676 31115 ts_tablet_manager.cc:1397] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.017s
I20250902 01:57:50.325536 31343 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.326464 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:50.327622 30892 leader_election.cc:343] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.332381 30965 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.331171 31157 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.333099 31146 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.333544 31146 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:50.333902 31157 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.334292 31146 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.335512 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:50.337393 30892 leader_election.cc:343] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.321921 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.341303 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.341945 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:50.342921 30926 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:50.345467 31118 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:50.345278 31115 raft_consensus.cc:357] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.345870 31115 raft_consensus.cc:383] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.346000 31118 ts_tablet_manager.cc:1397] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.079s user 0.016s sys 0.008s
I20250902 01:57:50.346122 31115 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.347471 31039 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.348810 31377 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.349148 31377 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.349089 31118 raft_consensus.cc:357] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.349543 31377 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.349813 31118 raft_consensus.cc:383] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.346776 31115 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:50.351423 30889 leader_election.cc:343] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.351823 31116 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.351887 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.352355 31116 ts_tablet_manager.cc:1397] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.008s
I20250902 01:57:50.352389 31115 ts_tablet_manager.cc:1428] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.028s user 0.002s sys 0.003s
I20250902 01:57:50.352782 31343 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.353335 31115 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.354641 31116 raft_consensus.cc:357] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.355288 31116 raft_consensus.cc:383] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.355587 31116 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.350139 31118 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.356452 31118 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.358433 31118 ts_tablet_manager.cc:1428] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250902 01:57:50.342442 31001 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.358928 31115 tablet_bootstrap.cc:654] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.358857 31377 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.365438 31001 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:50.366395 31116 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.371996 31116 ts_tablet_manager.cc:1428] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250902 01:57:50.372781 31116 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.375697 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.378655 31343 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.379282 31343 raft_consensus.cc:695] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:50.380290 31343 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.383497 30823 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.383059 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.378364 31116 tablet_bootstrap.cc:654] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.392182 31118 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:50.387199 30922 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.423338 30922 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:50.441077 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:50.444350 31038 leader_election.cc:343] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.441735 31377 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.446283 31039 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.447808 31377 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.448516 31377 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:50.449622 31377 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.457756 31116 tablet_bootstrap.cc:492] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.458180 31116 ts_tablet_manager.cc:1397] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.030s
I20250902 01:57:50.465413 30823 catalog_manager.cc:5582] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:50.471410 31343 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.472047 31343 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.474109 31343 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.474285 31116 raft_consensus.cc:357] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.474617 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.474675 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.474918 31116 raft_consensus.cc:383] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.475152 31116 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
W20250902 01:57:50.476300 30892 leader_election.cc:343] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.476787 31116 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.480409 31118 tablet_bootstrap.cc:654] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:50.480937 30889 leader_election.cc:343] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.481000 31115 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.482725 31116 ts_tablet_manager.cc:1428] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.024s user 0.006s sys 0.002s
I20250902 01:57:50.482668 31115 ts_tablet_manager.cc:1397] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.130s user 0.045s sys 0.070s
I20250902 01:57:50.483304 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.483974 31343 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.483770 31116 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.489831 31115 raft_consensus.cc:357] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.490471 31115 raft_consensus.cc:383] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.490864 31115 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.490417 30823 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:50.491637 31116 tablet_bootstrap.cc:654] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.492439 31115 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.494982 31376 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.495388 31376 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.497246 31376 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.498528 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.499301 31115 ts_tablet_manager.cc:1428] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250902 01:57:50.500176 31115 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.501495 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.502619 31376 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.503079 31376 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.506240 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.506770 30997 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:50.507711 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:50.508431 31215 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.508780 31215 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.508842 31039 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.509187 31215 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.509621 31344 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.509445 31116 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.510377 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.510993 31116 ts_tablet_manager.cc:1397] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.006s
I20250902 01:57:50.511111 30926 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:50.512075 31376 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.513873 31116 raft_consensus.cc:357] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.514873 31116 raft_consensus.cc:383] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.515129 31116 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.515771 31116 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.517330 31215 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.520045 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:50.520325 31118 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:50.520608 30922 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.520757 31118 ts_tablet_manager.cc:1397] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.129s user 0.022s sys 0.013s
I20250902 01:57:50.520502 31215 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.521742 31116 ts_tablet_manager.cc:1428] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250902 01:57:50.522789 31116 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.523069 31115 tablet_bootstrap.cc:654] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.523365 31118 raft_consensus.cc:357] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.524029 31118 raft_consensus.cc:383] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.524253 31118 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.524832 31118 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.528195 30922 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:50.533142 31039 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.536347 31376 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:50.536854 31376 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:50.536947 31344 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.537331 31344 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.537650 31376 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.539173 31344 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.541570 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.570713 30922 raft_consensus.cc:2391] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:50.571789 31118 ts_tablet_manager.cc:1428] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.051s user 0.006s sys 0.001s
I20250902 01:57:50.572549 31118 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:50.540726 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.577526 31039 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.578497 31377 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.579967 31118 tablet_bootstrap.cc:654] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.583961 31116 tablet_bootstrap.cc:654] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.584681 31115 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.585054 31115 ts_tablet_manager.cc:1397] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.015s
I20250902 01:57:50.587426 31115 raft_consensus.cc:357] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.590044 31115 raft_consensus.cc:383] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.590328 31115 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.590986 31115 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.592326 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:50.585301 31377 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.595908 30997 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.605854 30823 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:50.609635 31115 ts_tablet_manager.cc:1428] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.024s user 0.003s sys 0.004s
I20250902 01:57:50.613749 31115 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.618197 30996 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.622452 30997 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:50.636452 31185 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.639207 31157 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.641479 31343 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.657824 31115 tablet_bootstrap.cc:654] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.649271 31343 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250902 01:57:50.669624 31118 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:50.670033 31118 ts_tablet_manager.cc:1397] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.006s
I20250902 01:57:50.673636 31116 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:50.674155 31116 ts_tablet_manager.cc:1397] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.152s user 0.033s sys 0.042s
I20250902 01:57:50.696745 31343 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.707475 31378 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250902 01:57:50.714093 30967 consensus_peers.cc:489] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 -> Peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Couldn't send request to peer 088424002a894fd5bcc11a06af111445. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:50.709825 31343 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.717505 31343 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.717798 31345 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.718397 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.720534 31072 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.721042 31118 raft_consensus.cc:357] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.721715 31118 raft_consensus.cc:383] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.721863 31345 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.722024 31118 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.722374 31345 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.722711 31118 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.725243 31116 raft_consensus.cc:357] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.725871 31116 raft_consensus.cc:383] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.726140 31116 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.726660 31116 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:50.729014 30889 leader_election.cc:343] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.734539 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.785876 31118 ts_tablet_manager.cc:1428] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.116s user 0.007s sys 0.000s
I20250902 01:57:50.733273 31345 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.835202 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.840030 31116 ts_tablet_manager.cc:1428] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.166s user 0.007s sys 0.011s
I20250902 01:57:50.832219 31344 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.844730 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.844880 31344 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.845322 30924 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:50.846249 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:50.847883 31116 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:50.848867 31115 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:50.849349 31115 ts_tablet_manager.cc:1397] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.238s user 0.015s sys 0.016s
I20250902 01:57:50.853377 31377 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.853735 31377 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.854028 31344 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:50.855036 31344 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.854081 31377 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.857911 31118 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:50.841886 31319 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.870018 31319 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.874061 31378 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.874859 31319 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.877194 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:50.878001 31038 leader_election.cc:343] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.879128 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:50.882619 31115 raft_consensus.cc:357] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.883450 31115 raft_consensus.cc:383] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:50.883940 31185 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.883788 31115 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:50.886600 31157 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.886976 31075 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.894048 31418 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
W20250902 01:57:50.897663 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:50.920334 31376 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:50.886308 31115 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.926501 31115 ts_tablet_manager.cc:1428] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.077s user 0.005s sys 0.001s
I20250902 01:57:50.927248 31115 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:50.928845 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.937894 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:50.950789 30922 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:50.939087 31376 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:50.946643 31377 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.947152 31424 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:50.955888 31184 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:50.956341 31184 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:50.958250 31184 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:50.962217 31116 tablet_bootstrap.cc:654] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.963943 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:50.964020 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.971982 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:50.972620 31072 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
W20250902 01:57:50.973870 31038 consensus_peers.cc:489] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:50.976267 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 136931us
W20250902 01:57:50.977090 30892 leader_election.cc:343] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.977171 31215 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:50.977487 30892 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:50.978340 31426 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:50.981923 31377 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:50.965781 30967 leader_election.cc:343] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:50.986163 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:50.986761 31424 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.987128 31424 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.987435 31424 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:50.969010 31118 tablet_bootstrap.cc:654] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:50.997587 30967 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:50.998817 31378 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:50.999418 31378 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:50.999784 31378 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:51.001132 31038 leader_election.cc:343] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.001825 31115 tablet_bootstrap.cc:654] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:51.003057 30892 consensus_peers.cc:489] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Couldn't send request to peer 088424002a894fd5bcc11a06af111445. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:51.005998 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.006161 31215 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:51.009204 31424 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.010811 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.023309 30923 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.015836 31426 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:57:51.017716 30965 leader_election.cc:343] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.036074 31426 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.063091 31424 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.089780 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:51.091771 30967 leader_election.cc:343] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.092295 31420 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.096160 31420 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.098294 31118 tablet_bootstrap.cc:492] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.098881 31118 ts_tablet_manager.cc:1397] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.241s user 0.017s sys 0.016s
I20250902 01:57:51.109303 31420 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.109969 31116 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.110376 31116 ts_tablet_manager.cc:1397] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.263s user 0.023s sys 0.020s
I20250902 01:57:51.111666 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:51.112897 31039 leader_election.cc:343] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.112746 31116 raft_consensus.cc:357] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.113365 31116 raft_consensus.cc:383] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.113613 31116 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.113579 31426 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.114133 31118 raft_consensus.cc:357] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.114701 31118 raft_consensus.cc:383] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.114351 31116 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.114905 31118 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.115463 31118 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.116730 31116 ts_tablet_manager.cc:1428] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250902 01:57:51.117471 31116 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.133162 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.135068 31118 ts_tablet_manager.cc:1428] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.036s user 0.003s sys 0.003s
I20250902 01:57:51.135907 31118 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.133993 30921 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.138497 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.132184 31378 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.148766 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:51.152562 31038 leader_election.cc:343] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.153076 31038 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:51.164424 31115 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
W20250902 01:57:51.164752 30892 leader_election.cc:343] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.164853 31115 ts_tablet_manager.cc:1397] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.238s user 0.028s sys 0.042s
I20250902 01:57:51.154965 31344 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:51.187158 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.187670 31072 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.187654 31115 raft_consensus.cc:357] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.188244 31115 raft_consensus.cc:383] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.188418 31115 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.188840 31115 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.190184 31115 ts_tablet_manager.cc:1428] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250902 01:57:51.191720 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.195997 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.202754 30923 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:51.203799 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:51.204483 31377 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.204938 31377 raft_consensus.cc:695] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:51.205653 31377 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.206915 31115 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
W20250902 01:57:51.207326 30965 leader_election.cc:343] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.180145 31378 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:51.220588 30889 leader_election.cc:343] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.223838 30889 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:51.224750 31418 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250902 01:57:51.235348 31439 log.cc:927] Time spent T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250902 01:57:51.241110 30921 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.230408 31072 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.254431 30967 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:51.260118 31185 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.263007 31185 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:51.263804 31185 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.265139 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:51.266491 31319 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.266929 31319 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:51.267948 31376 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.267720 31319 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.268543 31376 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.270808 31376 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.273078 31116 tablet_bootstrap.cc:654] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.280071 31072 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.282608 31118 tablet_bootstrap.cc:654] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.302699 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.303665 31115 tablet_bootstrap.cc:654] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.306445 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.308815 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:51.311272 30822 catalog_manager.cc:5582] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:51.339092 31116 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.339092 31118 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.339577 31116 ts_tablet_manager.cc:1397] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.222s user 0.013s sys 0.011s
I20250902 01:57:51.339577 31118 ts_tablet_manager.cc:1397] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.204s user 0.020s sys 0.018s
I20250902 01:57:51.341975 31116 raft_consensus.cc:357] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.342141 31118 raft_consensus.cc:357] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.342713 31118 raft_consensus.cc:383] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.342655 31116 raft_consensus.cc:383] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.343043 31118 raft_consensus.cc:738] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.343078 31116 raft_consensus.cc:738] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.343988 31116 consensus_queue.cc:260] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.343904 31118 consensus_queue.cc:260] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.346496 31116 ts_tablet_manager.cc:1428] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250902 01:57:51.347378 31116 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.347265 31115 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:51.348588 31115 ts_tablet_manager.cc:1397] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.142s user 0.022s sys 0.012s
I20250902 01:57:51.349256 31118 ts_tablet_manager.cc:1428] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250902 01:57:51.350106 31118 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.351656 31115 raft_consensus.cc:357] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.352394 31115 raft_consensus.cc:383] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.352649 31115 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.353245 31115 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.353801 31116 tablet_bootstrap.cc:654] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.354600 31319 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.355036 31319 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.355413 31115 ts_tablet_manager.cc:1428] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:51.356283 31115 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:51.356809 31319 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.356931 31118 tablet_bootstrap.cc:654] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.359766 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.360319 31074 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.362785 31116 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.363298 31116 ts_tablet_manager.cc:1397] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20250902 01:57:51.363736 31115 tablet_bootstrap.cc:654] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.365614 31116 raft_consensus.cc:357] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.366020 31118 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.366204 31116 raft_consensus.cc:383] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.366415 31118 ts_tablet_manager.cc:1397] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20250902 01:57:51.367940 31345 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:51.366432 31116 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.368219 31118 raft_consensus.cc:357] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.368968 31118 raft_consensus.cc:383] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.369278 31118 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.368897 31116 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.369858 31118 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.371183 31116 ts_tablet_manager.cc:1428] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250902 01:57:51.371869 31118 ts_tablet_manager.cc:1428] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250902 01:57:51.372023 31116 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.372659 31118 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.373190 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.373838 30925 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.376950 30967 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.378795 31118 tablet_bootstrap.cc:654] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.379889 31116 tablet_bootstrap.cc:654] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.379720 31185 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.380306 31185 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.380713 31185 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.391384 31118 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.391798 31118 ts_tablet_manager.cc:1397] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.011s
I20250902 01:57:51.393580 31116 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.393899 31116 ts_tablet_manager.cc:1397] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.008s
I20250902 01:57:51.393926 31118 raft_consensus.cc:357] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.394549 31118 raft_consensus.cc:383] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.394784 31118 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.395633 31118 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.397178 31116 raft_consensus.cc:357] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.397809 31116 raft_consensus.cc:383] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.397840 31118 ts_tablet_manager.cc:1428] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:57:51.398181 31116 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.398859 31118 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.398834 31116 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.404467 31185 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.407425 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.408108 30925 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.408735 31116 ts_tablet_manager.cc:1428] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.015s user 0.005s sys 0.003s
I20250902 01:57:51.409914 31116 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.417733 31185 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.419198 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.419828 31118 tablet_bootstrap.cc:654] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.425616 31072 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.421072 30820 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.433722 31115 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:51.434028 31072 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.434159 31115 ts_tablet_manager.cc:1397] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.050s
I20250902 01:57:51.435616 30925 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.436106 31116 tablet_bootstrap.cc:654] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.436053 30967 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.445612 31116 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.445624 31118 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.446087 31116 ts_tablet_manager.cc:1397] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.004s
I20250902 01:57:51.446139 31118 ts_tablet_manager.cc:1397] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.008s
I20250902 01:57:51.446944 31422 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.447407 31422 raft_consensus.cc:695] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:51.448354 31422 consensus_queue.cc:237] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.448537 31116 raft_consensus.cc:357] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.449191 31116 raft_consensus.cc:383] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.448839 31118 raft_consensus.cc:357] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.449432 31116 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.449719 31118 raft_consensus.cc:383] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.450019 31118 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.450125 31116 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.450614 31118 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.437294 31115 raft_consensus.cc:357] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.451730 31115 raft_consensus.cc:383] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.452059 31115 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.452270 31116 ts_tablet_manager.cc:1428] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250902 01:57:51.452378 31118 ts_tablet_manager.cc:1428] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250902 01:57:51.453116 31116 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.453184 31118 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.430064 30820 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.453086 31115 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.461607 31115 ts_tablet_manager.cc:1428] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20250902 01:57:51.462417 31115 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:51.464386 31116 tablet_bootstrap.cc:654] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.464386 31118 tablet_bootstrap.cc:654] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.468650 31422 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.469260 31422 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.471757 31422 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.472815 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.473440 30925 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.474467 30965 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.475910 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.476105 31422 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.476421 31422 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.476752 31422 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:51.477121 30967 leader_election.cc:343] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.475910 31115 tablet_bootstrap.cc:654] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.480208 31116 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.480603 31116 ts_tablet_manager.cc:1397] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.016s
I20250902 01:57:51.482023 31118 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.482513 31118 ts_tablet_manager.cc:1397] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.008s
I20250902 01:57:51.483080 31116 raft_consensus.cc:357] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.483750 31116 raft_consensus.cc:383] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.483991 31116 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.485395 31116 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.487434 31116 ts_tablet_manager.cc:1428] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250902 01:57:51.487138 31118 raft_consensus.cc:357] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.487830 31118 raft_consensus.cc:383] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.488103 31118 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.488243 31116 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.488797 31118 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.491098 31118 ts_tablet_manager.cc:1428] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250902 01:57:51.492048 31118 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.493700 31116 tablet_bootstrap.cc:654] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.498739 31118 tablet_bootstrap.cc:654] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.504891 31115 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:51.505697 31115 ts_tablet_manager.cc:1397] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.010s
I20250902 01:57:51.507174 31116 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.507826 31116 ts_tablet_manager.cc:1397] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20250902 01:57:51.507932 31115 raft_consensus.cc:357] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.508602 31115 raft_consensus.cc:383] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.508884 31115 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.509434 31115 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.510668 31116 raft_consensus.cc:357] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.511440 31116 raft_consensus.cc:383] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.511756 31116 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.512156 31115 ts_tablet_manager.cc:1428] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250902 01:57:51.512379 31116 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.512974 31115 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:51.514328 31116 ts_tablet_manager.cc:1428] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:51.515192 31116 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.518527 31115 tablet_bootstrap.cc:654] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.518633 31118 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.519099 31118 ts_tablet_manager.cc:1397] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.006s
I20250902 01:57:51.521268 31116 tablet_bootstrap.cc:654] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.521767 31118 raft_consensus.cc:357] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.522447 31118 raft_consensus.cc:383] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.522763 31118 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.527657 31118 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.530535 31118 ts_tablet_manager.cc:1428] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20250902 01:57:51.531224 31115 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:51.531426 31118 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.531912 31115 ts_tablet_manager.cc:1397] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20250902 01:57:51.532402 31116 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.532845 31116 ts_tablet_manager.cc:1397] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20250902 01:57:51.535092 31116 raft_consensus.cc:357] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.535874 31116 raft_consensus.cc:383] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.536140 31116 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.536944 31118 tablet_bootstrap.cc:654] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.536825 31116 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.538549 31116 ts_tablet_manager.cc:1428] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:57:51.539609 31116 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.543278 30822 catalog_manager.cc:5582] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.547387 31116 tablet_bootstrap.cc:654] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.548629 31118 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.549041 31118 ts_tablet_manager.cc:1397] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20250902 01:57:51.552094 31118 raft_consensus.cc:357] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.552892 31118 raft_consensus.cc:383] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.553201 31118 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.553926 31118 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.564950 31116 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.560308 31118 ts_tablet_manager.cc:1428] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250902 01:57:51.564220 31378 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.570200 31116 ts_tablet_manager.cc:1397] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.006s
I20250902 01:57:51.570567 31118 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.558007 31115 raft_consensus.cc:357] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.571319 31115 raft_consensus.cc:383] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.571604 31115 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.572238 31115 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.570163 31378 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.573104 31116 raft_consensus.cc:357] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.573796 31116 raft_consensus.cc:383] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.574091 31116 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.574673 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.574795 31116 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.575207 30925 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.576591 31118 tablet_bootstrap.cc:654] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.576856 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.577090 31116 ts_tablet_manager.cc:1428] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20250902 01:57:51.576613 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.577860 31074 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.578312 31116 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.579124 31378 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.580111 31146 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.580425 31146 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.580684 31146 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.585736 31116 tablet_bootstrap.cc:654] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.589522 31146 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.591712 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.592237 30925 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.486302 31422 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.605129 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.605597 30921 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.622767 31422 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.625412 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.626055 31072 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:51.640311 31038 consensus_peers.cc:489] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:51.640413 31116 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.640308 31072 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.641350 31116 ts_tablet_manager.cc:1397] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.063s user 0.025s sys 0.031s
I20250902 01:57:51.642204 30925 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.643309 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.644068 31422 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.644511 31422 raft_consensus.cc:695] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:51.645274 31422 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.645859 31116 raft_consensus.cc:357] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.646889 31116 raft_consensus.cc:383] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.647292 31116 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.648144 31116 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.648764 30967 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.646402 30921 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.649174 31146 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.649672 31146 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.647764 31419 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.653612 31419 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.655910 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.656399 31074 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.656591 31116 ts_tablet_manager.cc:1428] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.014s user 0.008s sys 0.003s
I20250902 01:57:51.658138 31116 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.661218 31418 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.661648 31418 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.663053 31345 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:51.668642 31115 ts_tablet_manager.cc:1428] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.136s user 0.003s sys 0.003s
I20250902 01:57:51.675240 31115 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:51.676436 31419 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.677419 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.677979 31070 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:51.678850 30892 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.679469 31419 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.679817 31419 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.680125 31419 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.687188 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.687948 31070 raft_consensus.cc:2391] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:51.687978 31115 tablet_bootstrap.cc:654] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.691874 31074 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:51.691903 31419 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.692947 31116 tablet_bootstrap.cc:654] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.695015 31146 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:51.695785 31146 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.698557 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.699071 31070 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.706142 31070 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:51.707015 30892 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.707681 31157 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.708053 31157 raft_consensus.cc:695] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:51.707906 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.708724 31157 consensus_queue.cc:237] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.711817 31116 tablet_bootstrap.cc:492] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.712428 31116 ts_tablet_manager.cc:1397] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.002s
I20250902 01:57:51.715304 31419 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.716426 31419 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.716820 31419 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.719566 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.720158 31070 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:51.721133 30892 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.721866 31157 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.722218 31157 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.722599 31157 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.728891 31157 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.731410 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.732277 31070 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.734992 30824 catalog_manager.cc:5582] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.736231 31157 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.715236 31116 raft_consensus.cc:357] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.737659 31116 raft_consensus.cc:383] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.737987 31116 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.738611 31419 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.738775 31116 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.740880 31116 ts_tablet_manager.cc:1428] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250902 01:57:51.741717 31070 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:51.742482 31116 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.742806 30892 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.743479 31380 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.743897 31380 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.745556 31380 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.745811 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.746057 31380 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.746543 31380 raft_consensus.cc:695] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:51.748822 31116 tablet_bootstrap.cc:654] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.748665 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:51.749182 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:51.749220 31073 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:51.750591 30892 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.747283 31380 consensus_queue.cc:237] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.757202 31380 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.758447 31380 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.759202 31380 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.759873 31070 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.765723 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.771209 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:51.771673 30824 catalog_manager.cc:5582] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.687292 31418 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.776346 31377 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.776808 31377 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.779068 31377 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.782116 31116 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.784126 31378 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.784837 31116 ts_tablet_manager.cc:1397] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.000s
I20250902 01:57:51.784797 31378 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.786583 31378 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.787321 31116 raft_consensus.cc:357] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.787982 31116 raft_consensus.cc:383] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.788216 31116 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.788928 31116 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.789985 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.791152 31116 ts_tablet_manager.cc:1428] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:57:51.791952 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:51.792130 31038 leader_election.cc:343] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.792541 31074 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:51.783744 31380 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.795962 30967 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.796597 31378 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:51.798564 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.799154 30993 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:51.799607 31380 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:51.800164 31422 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.800524 31422 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.800941 31422 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.802107 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.802569 31074 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.806843 31441 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.807230 31441 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.808874 31441 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.811601 31074 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:51.812973 30892 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:51.813928 31380 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:51.814535 31380 raft_consensus.cc:695] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:51.815534 31380 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.819043 30824 catalog_manager.cc:5582] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.824980 31185 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:51.833323 31116 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.823645 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.835896 31441 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.836355 31441 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.839748 31116 tablet_bootstrap.cc:654] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.839874 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.839325 30824 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
W20250902 01:57:51.842370 30889 leader_election.cc:343] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.845350 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:57:51.846127 30889 leader_election.cc:343] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.844138 31441 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.848227 31118 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:51.848641 31118 ts_tablet_manager.cc:1397] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.278s user 0.077s sys 0.067s
I20250902 01:57:51.850026 31422 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.852500 31422 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:51.851574 30889 leader_election.cc:343] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.852736 31118 raft_consensus.cc:357] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.853358 31118 raft_consensus.cc:383] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250902 01:57:51.853422 30889 leader_election.cc:343] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:51.853678 31118 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.854245 31118 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.855226 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:51.847766 31476 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.856220 31115 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:51.856613 31115 ts_tablet_manager.cc:1397] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.182s user 0.023s sys 0.028s
I20250902 01:57:51.855973 31073 raft_consensus.cc:2391] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:51.864791 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.865377 30925 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:51.865779 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.866250 30918 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:51.866588 31441 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.867856 31441 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.869225 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.870455 31441 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:51.870532 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.873087 31115 raft_consensus.cc:357] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.873517 31115 raft_consensus.cc:383] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.873736 31115 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.875125 31115 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.875312 30822 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:51.875782 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:51.876617 31118 ts_tablet_manager.cc:1428] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250902 01:57:51.877265 31118 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:51.881258 30993 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:51.886089 31116 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:51.886130 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:51.886844 31116 ts_tablet_manager.cc:1397] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.054s user 0.014s sys 0.005s
I20250902 01:57:51.878731 31441 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.891773 31377 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.895838 31115 ts_tablet_manager.cc:1428] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.039s user 0.005s sys 0.001s
I20250902 01:57:51.896311 31185 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:51.900329 31115 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:51.907696 31419 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:51.899430 31377 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:51.935248 31116 raft_consensus.cc:357] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.935997 31116 raft_consensus.cc:383] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:51.936306 31116 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:51.937008 31116 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:51.940443 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:51.914322 31419 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.901269 31441 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:51.935513 31377 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.953439 31118 tablet_bootstrap.cc:654] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:51.967463 31319 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:51.968448 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:51.972544 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:51.979930 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:51.993095 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:51.994278 31070 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:51.982144 31345 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:51.996330 31116 ts_tablet_manager.cc:1428] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.109s user 0.009s sys 0.016s
I20250902 01:57:51.997193 31116 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
I20250902 01:57:51.999195 31039 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:52.001562 30965 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:51.995884 31345 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:52.001123 31376 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:52.001036 31419 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:52.004355 30994 raft_consensus.cc:2391] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:52.010866 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:52.009969 31422 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250902 01:57:52.011824 31038 leader_election.cc:343] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.006068 31345 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.006160 31185 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:52.020819 31115 tablet_bootstrap.cc:654] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:52.006879 31376 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:52.021548 31476 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:52.010579 31441 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.037322 31378 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:52.037803 31378 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.041538 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:52.032079 31376 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:52.030995 31185 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:52.065378 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 65149.3us
I20250902 01:57:52.066699 31039 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:57:52.067668 31039 leader_election.cc:343] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.071568 31146 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:52.073339 31146 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250902 01:57:52.075220 30889 leader_election.cc:343] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.076777 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:52.076778 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.079448 31441 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:52.079908 31441 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.080523 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:52.080754 31185 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:52.081090 30965 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:52.081583 31441 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.074779 31378 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:52.088673 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:52.086050 31185 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:52.088104 31485 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:52.093099 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:52.093716 31485 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.095981 31073 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:52.098523 31345 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.095517 31485 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.102634 31377 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.104688 31377 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:52.108045 30892 leader_election.cc:343] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.108397 30892 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:57:52.112573 31038 leader_election.cc:343] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.119797 31380 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:52.120553 31118 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:52.121661 31118 ts_tablet_manager.cc:1397] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.245s user 0.012s sys 0.015s
I20250902 01:57:52.132315 30967 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:52.133606 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.134454 30924 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:52.135866 31146 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:52.136255 31146 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:52.136585 31146 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.137732 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:52.119935 31441 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:52.143904 31344 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:52.144268 31344 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:52.144618 31344 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.145661 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:57:52.150388 31039 leader_election.cc:343] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.160846 31118 raft_consensus.cc:357] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.161725 31118 raft_consensus.cc:383] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:52.162061 31118 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:52.151827 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.164280 31118 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.171301 31146 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.181869 31146 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:52.185920 31118 ts_tablet_manager.cc:1428] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.064s user 0.006s sys 0.002s
I20250902 01:57:52.149257 31441 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.189049 31115 tablet_bootstrap.cc:492] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:52.190734 31115 ts_tablet_manager.cc:1397] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.291s user 0.018s sys 0.011s
I20250902 01:57:52.168509 30922 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:52.189774 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.188232 31345 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.200254 30926 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 0.
I20250902 01:57:52.201040 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.201159 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.201644 30919 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:52.201738 30922 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.170832 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:52.205667 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 60186.5us
I20250902 01:57:52.206665 31146 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:52.213227 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:52.213738 31074 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.214156 31039 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:52.216595 31001 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.211922 30995 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.220084 31039 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:52.233135 31118 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:52.219967 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.239056 31487 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:52.243080 31074 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:52.246268 30967 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:52.247614 31146 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:52.248039 31146 raft_consensus.cc:695] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:52.248700 31146 consensus_queue.cc:237] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.249046 30924 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.245976 31485 raft_consensus.cc:2762] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250902 01:57:52.263233 31116 tablet_bootstrap.cc:654] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:52.275964 31492 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20250902 01:57:52.279137 31499 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:52.279470 31499 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:52.279796 31499 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.285396 31115 raft_consensus.cc:357] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.286206 31115 raft_consensus.cc:383] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:52.286801 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:52.287413 30925 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:52.287648 31441 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.289774 31115 raft_consensus.cc:738] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:52.292106 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:52.291397 31115 consensus_queue.cc:260] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.260497 31422 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20250902 01:57:52.258252 31376 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.304589 31377 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:52.326874 31115 ts_tablet_manager.cc:1428] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.136s user 0.008s sys 0.000s
I20250902 01:57:52.387020 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:52.335567 31377 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:52.376271 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:52.405493 31038 leader_election.cc:343] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:52.437193 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 109989us
I20250902 01:57:52.401376 31377 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.388904 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:52.454159 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 142742us
W20250902 01:57:52.461225 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 55758.5us
W20250902 01:57:52.469376 31038 leader_election.cc:343] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.488142 30924 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:52.470526 31418 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.064s
I20250902 01:57:52.545202 31118 tablet_bootstrap.cc:654] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:52.555334 31506 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20250902 01:57:52.539654 31376 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.561524 30922 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:52.566337 31115 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:52.558317 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.546182 31344 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.600612 31344 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.601068 31116 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:52.601523 31116 ts_tablet_manager.cc:1397] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.604s user 0.009s sys 0.013s
I20250902 01:57:52.596832 30996 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.546803 31508 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
W20250902 01:57:52.603742 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 63935.2us
I20250902 01:57:52.606451 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:52.661557 31377 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:52.662228 31511 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20250902 01:57:52.664830 31377 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.667578 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:57:52.671310 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 76839.6us
W20250902 01:57:52.672407 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 49796.6us
W20250902 01:57:52.693830 30889 consensus_peers.cc:489] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:52.744565 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 49685.3us
I20250902 01:57:52.815474 31116 raft_consensus.cc:357] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.817124 31116 raft_consensus.cc:383] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:52.817409 31116 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:52.803964 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:52.818674 31116 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.821493 31038 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:52.812688 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.818640 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.848726 31115 tablet_bootstrap.cc:654] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:52.828516 31485 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:57:52.830529 30919 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.853127 30918 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.869436 31116 ts_tablet_manager.cc:1428] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.268s user 0.016s sys 0.014s
I20250902 01:57:52.872922 31118 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:52.873343 31118 ts_tablet_manager.cc:1397] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.641s user 0.038s sys 0.077s
I20250902 01:57:52.837163 31516 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.095s
I20250902 01:57:52.874670 31497 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
W20250902 01:57:52.875770 30995 consensus_meta.cc:287] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.642s user 0.006s sys 0.014s
I20250902 01:57:52.876091 30995 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:52.868094 31066 raft_consensus.cc:1238] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:57:52.837162 31515 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.101s
I20250902 01:57:52.882289 31118 raft_consensus.cc:357] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:52.883371 31038 leader_election.cc:343] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:52.888377 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:52.891127 31118 raft_consensus.cc:383] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:52.891444 31118 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
W20250902 01:57:52.879963 31499 consensus_meta.cc:287] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.012s
I20250902 01:57:52.893182 31118 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.893252 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:52.896579 31497 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:52.898720 31500 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:52.898911 31499 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.899223 31500 raft_consensus.cc:695] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:52.900791 31499 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:52.905275 31118 ts_tablet_manager.cc:1428] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250902 01:57:52.909446 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:52.909948 30923 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:52.912561 31185 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:52.897392 31497 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.944682 31118 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Bootstrap starting.
I20250902 01:57:52.915002 31500 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:52.984169 31146 consensus_queue.cc:1046] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250902 01:57:52.988796 31038 leader_election.cc:343] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:53.005170 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:53.007421 31146 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:53.009124 31146 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250902 01:57:53.015417 31038 leader_election.cc:343] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:53.064642 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:53.001230 31424 raft_consensus.cc:3053] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:53.069697 31424 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:57:53.069885 31523 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20250902 01:57:53.066509 31515 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:53.075150 31377 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:53.030747 30923 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:53.099246 31039 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:53.079028 31377 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.078768 31515 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.078931 31424 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.090700 30919 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:53.105170 31344 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:53.160315 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:53.172968 31116 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Bootstrap starting.
W20250902 01:57:53.175168 31511 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20250902 01:57:53.137328 31525 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250902 01:57:53.230569 31486 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:53.177876 31344 raft_consensus.cc:695] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:53.231253 31486 raft_consensus.cc:695] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:53.249907 31515 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:53.257726 31115 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:53.258308 31115 ts_tablet_manager.cc:1397] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.692s user 0.026s sys 0.056s
W20250902 01:57:53.274871 31535 log.cc:927] Time spent T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
W20250902 01:57:53.306965 31528 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20250902 01:57:53.310628 31377 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:53.332067 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 171872us
I20250902 01:57:53.252758 30918 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:53.355062 30824 catalog_manager.cc:5582] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:53.357805 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:53.358382 30918 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:53.363903 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:57:53.371708 31534 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.106s user 0.003s sys 0.000s
I20250902 01:57:53.331096 31344 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.353544 31424 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:53.377782 31118 tablet_bootstrap.cc:654] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:53.382107 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:53.382728 31073 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:53.379082 31039 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:53.386191 31485 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:53.386615 31485 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:53.386976 31485 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:53.368245 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 165) took 1074 ms. Trace:
I20250902 01:57:53.363660 31486 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.379088 31499 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:53.448915 31116 tablet_bootstrap.cc:654] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:53.453938 31522 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250902 01:57:53.482465 31485 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.437880 30920 rpcz_store.cc:276] 0902 01:57:52.281890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:52.283045 (+ 1155us) service_pool.cc:225] Handling call
0902 01:57:52.325379 (+ 42334us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:57:52.461274 (+135895us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:52.465478 (+ 4204us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:57:52.489312 (+ 23834us) log.cc:844] Serialized 26 byte log entry
0902 01:57:52.730323 (+241011us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc5ff1b139c 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb
0902 01:57:52.826490 (+ 96167us) raft_consensus.cc:1643] Marking committed up to 0
0902 01:57:52.833176 (+ 6686us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:52.837566 (+ 4390us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:57:53.332649 (+495083us) raft_consensus.cc:1683] finished
0902 01:57:53.336512 (+ 3863us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:53.354227 (+ 17715us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76379,"raft.queue_time_us":81826,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":5772,"spinlock_wait_cycles":44608896,"thread_start_us":334058,"threads_started":1,"wal-append.queue_time_us":370024}
I20250902 01:57:53.488467 31115 raft_consensus.cc:357] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.489112 31115 raft_consensus.cc:383] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:53.489380 31115 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:53.489631 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:53.489938 31115 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:53.490656 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 322569us
I20250902 01:57:53.472481 31499 raft_consensus.cc:695] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:53.519542 31115 ts_tablet_manager.cc:1428] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.261s user 0.002s sys 0.011s
I20250902 01:57:53.556942 31485 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:53.685688 31499 consensus_queue.cc:237] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.756304 31512 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:53.756974 31512 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.768183 31512 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:53.769591 31545 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.267s user 0.001s sys 0.000s
I20250902 01:57:53.776083 31118 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: No bootstrap required, opened a new log
I20250902 01:57:53.776655 31118 ts_tablet_manager.cc:1397] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent bootstrapping tablet: real 0.832s user 0.058s sys 0.034s
I20250902 01:57:53.777702 31115 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
W20250902 01:57:53.779131 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 266951us
I20250902 01:57:53.786437 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:53.787694 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:53.793555 30995 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:53.799798 31118 raft_consensus.cc:357] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.800601 31118 raft_consensus.cc:383] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:53.800865 31118 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Initialized, Role: FOLLOWER
I20250902 01:57:53.801461 31118 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.811889 31118 ts_tablet_manager.cc:1428] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
W20250902 01:57:53.824414 31038 leader_election.cc:343] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
W20250902 01:57:53.825230 31038 leader_election.cc:343] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:53.836601 31543 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250902 01:57:53.847636 31116 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: No bootstrap required, opened a new log
I20250902 01:57:53.848028 31116 ts_tablet_manager.cc:1397] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Time spent bootstrapping tablet: real 0.675s user 0.017s sys 0.037s
I20250902 01:57:53.863504 31115 tablet_bootstrap.cc:654] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:53.876188 31116 raft_consensus.cc:357] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:53.877851 31116 raft_consensus.cc:383] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:53.878274 31116 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Initialized, Role: FOLLOWER
I20250902 01:57:53.879101 31116 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.088611 31486 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.417479 31116 ts_tablet_manager.cc:1428] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Time spent starting tablet: real 0.569s user 0.008s sys 0.013s
W20250902 01:57:54.417778 31116 ts_tablet_manager.cc:1460] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Tablet startup took 569ms
I20250902 01:57:54.424182 31115 tablet_bootstrap.cc:492] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:54.424773 31115 ts_tablet_manager.cc:1397] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.647s user 0.012s sys 0.011s
I20250902 01:57:54.355669 31441 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.372869 31376 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:57:54.484161 31527 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250902 01:57:54.402623 31476 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.236116 31486 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.414533 31512 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:57:54.418067 31116 ts_tablet_manager.cc:1462] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Trace:
0902 01:57:45.435084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:45.806072 (+370988us) service_pool.cc:225] Handling call
0902 01:57:45.836065 (+ 29993us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0902 01:57:45.838673 (+ 2608us) ts_tablet_manager.cc:676] Creating new metadata...
0902 01:57:45.971887 (+133214us) tablet_metadata.cc:641] Metadata flushed
0902 01:57:46.043523 (+ 71636us) tablet_replica.cc:206] Creating consensus instance
0902 01:57:46.067997 (+ 24474us) inbound_call.cc:173] Queueing success response
0902 01:57:53.172674 (+7104677us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0902 01:57:53.848245 (+675571us) ts_tablet_manager.cc:1429] Starting tablet replica
0902 01:57:53.867980 (+ 19735us) tablet_replica.cc:260] Starting instrumentation
0902 01:57:53.868569 (+ 589us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":12858,"raft.queue_time_us":96282,"spinlock_wait_cycles":61035392,"tablet-open.queue_time_us":7104769}
I20250902 01:57:54.516353 31476 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.512997 31376 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.495342 31441 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.516593 31512 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.605983 31497 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:54.618990 31492 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.620635 31476 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.637542 31476 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.638100 31476 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.639129 31512 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.639585 31512 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.639930 31512 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.640115 31476 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:54.640518 31558 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250902 01:57:54.641657 31512 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.640152 31115 raft_consensus.cc:357] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.641927 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 226) took 1266 ms. Trace:
I20250902 01:57:54.620347 31497 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.642117 31115 raft_consensus.cc:383] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:54.642526 31115 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:54.643410 31115 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.642324 31067 rpcz_store.cc:276] 0902 01:57:53.376183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:53.523968 (+147785us) service_pool.cc:225] Handling call
0902 01:57:53.524324 (+ 356us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:57:53.796016 (+271692us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:53.796045 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:57:53.825317 (+ 29272us) log.cc:844] Serialized 26 byte log entry
0902 01:57:54.059309 (+233992us) raft_consensus.cc:1643] Marking committed up to 0
0902 01:57:54.086661 (+ 27352us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:54.088340 (+ 1679us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:57:54.641709 (+553369us) raft_consensus.cc:1683] finished
0902 01:57:54.641726 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:54.641852 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":268658,"raft.queue_time_us":271658,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7438,"thread_start_us":44420,"threads_started":1,"wal-append.queue_time_us":682880}
I20250902 01:57:54.649482 31376 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.634728 31499 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.650177 31115 ts_tablet_manager.cc:1428] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.225s user 0.004s sys 0.004s
I20250902 01:57:54.650238 31499 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:54.654035 31000 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.002s sys 0.002s
W20250902 01:57:54.654361 31000 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.002s sys 0.002s
I20250902 01:57:54.643885 31564 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:54.643497 31497 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.652063 31500 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.664949 31441 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.665381 31376 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.679663 31504 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.679788 31376 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.680104 31504 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.656976 31185 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:54.681944 31185 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.683983 31185 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.683350 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:54.685042 31038 consensus_peers.cc:489] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:57:54.685467 31555 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.817s user 0.010s sys 0.006s
I20250902 01:57:54.673445 31564 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.698302 31564 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.652271 31492 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.675552 31500 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.679329 31576 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.702201 31576 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.707945 31492 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.709555 31576 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.710865 31376 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.723963 31567 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.724498 31567 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.726528 31567 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.723878 31578 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.727265 31578 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.730370 31578 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.751004 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 191) took 1253 ms. Trace:
I20250902 01:57:54.751464 30919 rpcz_store.cc:276] 0902 01:57:53.497179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:53.502833 (+ 5654us) service_pool.cc:225] Handling call
0902 01:57:54.750196 (+1247363us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:57:54.750404 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:54.750433 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:57:54.750483 (+ 50us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:57:54.750516 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:54.750763 (+ 247us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:54.750915 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:54.765676 31578 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.766387 31578 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.768465 31578 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.778254 31578 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.778769 31578 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.780635 31578 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.781600 31185 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.782125 31185 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.786026 31185 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.793148 31500 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:57:54.797154 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:54.833225 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.834141 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.835274 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.836623 31575 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.837177 31567 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.837297 31575 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.837672 31567 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.839167 31575 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.839629 31575 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.840010 31575 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.841734 31575 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.842170 31575 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.842577 31575 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.843657 31476 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.844095 31476 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.845750 31476 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.846169 31476 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:54.846573 31476 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.851817 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.839174 31567 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.853255 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:54.853793 30918 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:54.854050 31504 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.855093 31476 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.857812 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.858783 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.860544 30918 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:54.861740 31575 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.863433 31115 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:54.864027 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.866233 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.867085 30922 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:54.868693 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.870108 31567 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.870584 31567 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.872407 31567 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.872828 31567 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.873270 31567 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.875602 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 210) took 1005 ms. Trace:
I20250902 01:57:54.876055 31476 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.875963 31000 rpcz_store.cc:276] 0902 01:57:53.870718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:53.876605 (+ 5887us) service_pool.cc:225] Handling call
0902 01:57:53.890403 (+ 13798us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:57:53.890721 (+ 318us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:53.890755 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:57:53.893328 (+ 2573us) log.cc:844] Serialized 26 byte log entry
0902 01:57:54.874685 (+981357us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b18000ff760. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171
0902 01:57:54.874829 (+ 144us) raft_consensus.cc:1643] Marking committed up to 0
0902 01:57:54.874870 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:54.874948 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:57:54.874995 (+ 47us) raft_consensus.cc:1683] finished
0902 01:57:54.875012 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:54.875523 (+ 511us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":382,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":6247,"spinlock_wait_cycles":2301184,"thread_start_us":760929,"threads_started":1,"wal-append.queue_time_us":760769}
I20250902 01:57:54.880300 31476 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.882643 31476 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.885987 31476 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:54.886503 31476 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.890094 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 192) took 1083 ms. Trace:
I20250902 01:57:54.890509 31072 rpcz_store.cc:276] 0902 01:57:53.806978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:54.889313 (+1082335us) service_pool.cc:225] Handling call
0902 01:57:54.889603 (+ 290us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:57:54.889766 (+ 163us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:54.889798 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:57:54.889845 (+ 47us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:57:54.889883 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:54.889940 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:54.890041 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:54.901664 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:54.904970 31567 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.905488 31567 raft_consensus.cc:2762] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250902 01:57:54.902962 31578 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:54.906414 31578 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:54.908089 31115 tablet_bootstrap.cc:654] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:54.908180 31578 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:54.912671 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 198) took 1127 ms. Trace:
I20250902 01:57:54.913153 30996 rpcz_store.cc:276] 0902 01:57:53.785544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:54.911650 (+1126106us) service_pool.cc:225] Handling call
0902 01:57:54.912039 (+ 389us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:57:54.912253 (+ 214us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:54.912292 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:57:54.912351 (+ 59us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:57:54.912406 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:54.912483 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:54.912607 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:57:54.920073 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:54.920737 30992 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:54.930569 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.931178 30919 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:54.934160 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.938916 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.941195 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.944165 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.945114 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.944780 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.954301 31115 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:54.955006 31115 ts_tablet_manager.cc:1397] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.008s
I20250902 01:57:54.955116 30924 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
W20250902 01:57:54.958125 30889 leader_election.cc:343] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:54.972173 31499 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:54.995411 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.995788 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.997916 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:54.998986 30920 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:55.003806 31592 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.004246 31592 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.006182 31592 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.012287 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.014127 31512 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.014559 31512 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.016355 31512 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:55.018256 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 332626us
I20250902 01:57:55.022377 31586 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:55.022758 31586 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.024699 31586 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.025707 31586 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.026098 31586 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.027689 31586 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.028303 31476 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.030078 31486 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.035719 31499 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:57:55.035971 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 350575us
I20250902 01:57:55.036165 31499 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.037079 31039 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:55.038240 31576 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.037837 31499 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.038712 31576 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.038985 31543 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.038791 31499 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:55.039502 31543 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.040360 31576 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.041359 31543 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.041217 31572 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:55.042133 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.042436 31376 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:55.043476 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:55.043668 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 110371us
I20250902 01:57:55.043723 31499 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.039053 31345 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election won for term 1
I20250902 01:57:55.044638 31345 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:55.045423 31345 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.047885 31574 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.048353 31574 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.050501 31499 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.050068 30821 catalog_manager.cc:5582] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:55.050129 31574 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.052301 31499 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.051677 30821 catalog_manager.cc:5582] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:55.052734 31499 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.053326 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.053995 31070 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:55.053586 30821 catalog_manager.cc:5582] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:55.055615 31115 raft_consensus.cc:357] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.056197 31115 raft_consensus.cc:383] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:55.056417 31115 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:55.057192 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.057685 31568 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.058270 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.059303 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.058113 31568 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.056960 31115 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.060791 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.056717 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.062039 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.065697 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.066798 31519 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.067121 30993 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 0.
I20250902 01:57:55.066673 30821 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:55.067255 31519 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.069327 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:57:55.069753 30889 leader_election.cc:343] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:55.070271 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 204) took 1282 ms. Trace:
I20250902 01:57:55.070329 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.074209 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.075506 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.065228 31499 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.079389 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:55.078213 31557 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.064495 31568 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.064798 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.082968 31503 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.082880 31378 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.083856 30889 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:55.085433 30822 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44296 (request call id 110) took 1528 ms. Trace:
I20250902 01:57:55.078651 31564 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.086131 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.087461 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.071265 31519 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.073038 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.052301 31574 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.088666 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.088721 31485 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250902 01:57:55.089264 30824 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44308 (request call id 98) took 2269 ms. Trace:
I20250902 01:57:55.088996 31557 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.088740 31573 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.074285 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.091940 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:55.092262 30889 leader_election.cc:343] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:55.075239 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.093823 31039 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:57:55.104559 30889 leader_election.cc:343] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:55.106423 31570 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:57:55.106528 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:55.106920 31570 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:55.107384 31570 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:55.107847 31504 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:55.108979 31504 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:55.109176 31504 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:55.109385 31504 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
W20250902 01:57:55.112347 31038 consensus_peers.cc:489] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:55.096400 31115 ts_tablet_manager.cc:1428] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.141s user 0.002s sys 0.004s
I20250902 01:57:55.099457 31067 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:57:55.091693 30822 rpcz_store.cc:276] 0902 01:57:53.556968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:53.557136 (+ 168us) service_pool.cc:225] Handling call
0902 01:57:55.085370 (+1528234us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":345,"reported_tablets":7}
I20250902 01:57:55.090996 31503 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.118729 31441 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.119290 31441 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.093582 31378 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.122860 31441 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.123384 31115 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:55.123930 31378 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.124482 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.125442 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.130544 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.132299 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.132722 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.133682 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.134553 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.135411 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.136670 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.095222 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:55.101445 31573 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.095367 31345 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
W20250902 01:57:55.148658 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 69320.5us
I20250902 01:57:55.146957 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.131569 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.149931 31573 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.095554 31564 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.135746 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.097155 31574 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.139339 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.308763 31509 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:57:55.262478 31593 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20250902 01:57:55.418397 31509 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.419399 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.419055 31564 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.309092 31499 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:57:55.241281 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 88282.7us
I20250902 01:57:55.420516 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.330765 31519 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.420567 31499 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.189162 31424 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.421566 31519 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.308785 31115 tablet_bootstrap.cc:654] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:55.422142 31424 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.095841 30824 rpcz_store.cc:276] 0902 01:57:52.819061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:52.825102 (+ 6041us) service_pool.cc:225] Handling call
0902 01:57:55.082420 (+2257318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:57:53.473664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:57:53.487780 (+ 14116us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:57:53.487823 (+ 43us) write_op.cc:435] Acquired schema lock
0902 01:57:53.487847 (+ 24us) tablet.cc:660] Decoding operations
0902 01:57:53.488222 (+ 375us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:57:53.488308 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0902 01:57:53.488378 (+ 70us) tablet.cc:683] Acquiring locks for 1 operations
0902 01:57:53.488580 (+ 202us) tablet.cc:699] Row locks acquired
0902 01:57:53.488602 (+ 22us) write_op.cc:260] PREPARE: finished
0902 01:57:53.488758 (+ 156us) write_op.cc:270] Start()
0902 01:57:53.488892 (+ 134us) write_op.cc:276] Timestamp: P: 1756778273488708 usec, L: 0
0902 01:57:53.488914 (+ 22us) op_driver.cc:348] REPLICATION: starting
0902 01:57:53.489571 (+ 657us) log.cc:844] Serialized 540 byte log entry
0902 01:57:53.850836 (+361265us) op_driver.cc:464] REPLICATION: finished
0902 01:57:54.607653 (+756817us) write_op.cc:301] APPLY: starting
0902 01:57:54.610333 (+ 2680us) tablet.cc:1365] starting BulkCheckPresence
0902 01:57:54.615116 (+ 4783us) tablet.cc:1368] finished BulkCheckPresence
0902 01:57:54.619789 (+ 4673us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:57:54.625789 (+ 6000us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:57:54.636448 (+ 10659us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0902 01:57:54.637616 (+ 1168us) write_op.cc:312] APPLY: finished
0902 01:57:54.637798 (+ 182us) log.cc:844] Serialized 22 byte log entry
0902 01:57:54.637940 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:57:54.638032 (+ 92us) write_op.cc:454] Released schema lock
0902 01:57:54.638066 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0902 01:57:54.638368 (+ 302us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:57:54.638399 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":235686,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":679093,"apply.run_cpu_time_us":1874,"apply.run_wall_time_us":84680,"num_ops":1,"prepare.queue_time_us":95292,"prepare.run_cpu_time_us":6745,"prepare.run_wall_time_us":31619,"raft.queue_time_us":55657,"raft.run_cpu_time_us":95766,"raft.run_wall_time_us":254089,"replication_time_us":356797,"thread_start_us":262767,"threads_started":4,"wal-append.queue_time_us":282420}]]}
I20250902 01:57:55.423979 31424 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.250860 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.425921 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.427039 31574 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.070797 30995 rpcz_store.cc:276] 0902 01:57:53.787395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:53.793017 (+ 5622us) service_pool.cc:225] Handling call
0902 01:57:53.793381 (+ 364us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:57:54.473680 (+680299us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:57:54.495064 (+ 21384us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:57:54.529782 (+ 34718us) log.cc:844] Serialized 26 byte log entry
0902 01:57:54.677210 (+147428us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:57:55.001860 (+324650us) log.cc:844] Serialized 14 byte log entry
0902 01:57:55.069952 (+ 68092us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:55.070022 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:57:55.070065 (+ 43us) raft_consensus.cc:1683] finished
0902 01:57:55.070083 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:55.070187 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72395,"prepare.queue_time_us":45384,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":467755,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":31301,"spinlock_wait_cycles":2432,"thread_start_us":382704,"threads_started":2,"wal-append.queue_time_us":144061}
I20250902 01:57:55.427886 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.330725 31568 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.428436 31519 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.428921 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.428803 31568 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.429637 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.417647 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.431063 31568 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.330435 31565 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:55.135749 31588 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:55.104607 30994 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:57:55.432106 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:55.432427 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 113959us
I20250902 01:57:55.250835 31557 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.417577 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:55.228560 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 77982.1us
I20250902 01:57:55.241029 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.155900 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.436409 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.460170 31509 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.318550 31575 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:55.421586 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.440615 31499 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.753463 31564 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:55.697381 31573 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:55.245218 31503 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.844717 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.630694 31345 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:55.803858 31492 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:57:55.642681 31568 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:57:55.804163 31105 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.29.21.254:45755, user_credentials={real_user=slave}} blocked reactor thread for 171813us
I20250902 01:57:55.393651 31570 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.837884 31588 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:55.696496 31424 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:55.436024 31576 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:55.632937 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.418421 30918 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:55.460342 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:55.942538 31068 raft_consensus.cc:2391] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:55.893317 31424 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.944152 31115 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:55.882910 31345 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.945230 31115 ts_tablet_manager.cc:1397] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.832s user 0.024s sys 0.083s
I20250902 01:57:55.946590 31424 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.839257 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:55.899104 31504 consensus_meta.cc:287] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.789s user 0.012s sys 0.000s
I20250902 01:57:55.895576 31492 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:55.948802 31492 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:55.949016 31504 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.952212 31504 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.836773 31557 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:57:55.723273 30947 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0a80 after lost signal to thread 29786
I20250902 01:57:55.954069 31557 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:55.955600 30947 debug-util.cc:398] Leaking SignalData structure 0x7b0800199ba0 after lost signal to thread 30858
I20250902 01:57:55.956612 31557 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.856348 31564 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.845960 31575 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:55.957437 30947 debug-util.cc:398] Leaking SignalData structure 0x7b08002addc0 after lost signal to thread 30950
I20250902 01:57:55.882658 31573 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.899853 31570 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.960176 31575 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.961462 31573 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.962466 31564 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.883126 31568 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.461920 31565 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.891763 31588 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:55.964879 31568 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.972543 31345 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.966542 31565 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.974606 31115 raft_consensus.cc:357] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.975380 31115 raft_consensus.cc:383] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:55.975683 31115 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:55.975098 31588 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:55.978236 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 130489us
I20250902 01:57:55.978735 30917 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250902 01:57:55.985875 31588 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:55.985971 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.987071 31070 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:57:55.988368 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:55.896252 31576 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.995218 31576 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:55.997011 31492 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:55.998374 31115 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.001000 31492 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:57:55.995406 31038 consensus_peers.cc:489] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:56.009480 30892 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:57:56.011729 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 898190us
I20250902 01:57:56.012948 31610 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:56.013480 31610 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:56.017714 31610 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.005038 31115 ts_tablet_manager.cc:1428] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.059s user 0.007s sys 0.003s
I20250902 01:57:56.012979 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.022794 31377 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:56.023722 31377 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:57:56.025017 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 224590us
I20250902 01:57:56.026209 31377 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.024039 31115 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
W20250902 01:57:56.032615 31038 consensus_peers.cc:489] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:57:56.033879 31497 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.024082 30892 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.065141 30892 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.066905 30892 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.067766 31588 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.069818 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.074450 31611 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:57:56.084769 30947 debug-util.cc:398] Leaking SignalData structure 0x7b08002a37a0 after lost signal to thread 31025
W20250902 01:57:56.088624 30947 debug-util.cc:398] Leaking SignalData structure 0x7b0800263340 after lost signal to thread 31099
I20250902 01:57:56.088502 30892 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:57:56.089012 30947 debug-util.cc:398] Leaking SignalData structure 0x7b0800246f20 after lost signal to thread 31100
I20250902 01:57:56.089953 30892 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.090193 31612 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.090786 31612 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.095399 31578 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.095535 31610 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.091217 31604 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.099484 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.091329 30892 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.102236 31617 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.102319 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.104619 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.112263 31509 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.112854 31509 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.116508 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.117420 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.114919 31509 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.118157 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.119987 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:57:56.128365 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 247875us
I20250902 01:57:56.123350 30821 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44316 (request call id 104) took 1229 ms. Trace:
I20250902 01:57:56.100992 31612 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.135738 31617 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.132098 31565 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.138733 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.140270 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.132877 30821 rpcz_store.cc:276] 0902 01:57:54.893799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:54.893941 (+ 142us) service_pool.cc:225] Handling call
0902 01:57:56.123022 (+1229081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:57:55.104788 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:57:55.105557 (+ 769us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:57:55.105596 (+ 39us) write_op.cc:435] Acquired schema lock
0902 01:57:55.105615 (+ 19us) tablet.cc:660] Decoding operations
0902 01:57:55.264245 (+158630us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:57:55.264339 (+ 94us) write_op.cc:641] Partition lock acquired for write op
0902 01:57:55.264378 (+ 39us) tablet.cc:683] Acquiring locks for 4 operations
0902 01:57:55.264906 (+ 528us) tablet.cc:699] Row locks acquired
0902 01:57:55.264931 (+ 25us) write_op.cc:260] PREPARE: finished
0902 01:57:55.265103 (+ 172us) write_op.cc:270] Start()
0902 01:57:55.265228 (+ 125us) write_op.cc:276] Timestamp: P: 1756778275265054 usec, L: 0
0902 01:57:55.265254 (+ 26us) op_driver.cc:348] REPLICATION: starting
0902 01:57:55.266155 (+ 901us) log.cc:844] Serialized 1705 byte log entry
0902 01:57:55.896056 (+629901us) op_driver.cc:464] REPLICATION: finished
0902 01:57:55.980464 (+ 84408us) write_op.cc:301] APPLY: starting
0902 01:57:55.980640 (+ 176us) tablet.cc:1365] starting BulkCheckPresence
0902 01:57:55.980896 (+ 256us) tablet.cc:1368] finished BulkCheckPresence
0902 01:57:55.981103 (+ 207us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:57:55.981861 (+ 758us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:57:55.982263 (+ 402us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0902 01:57:55.982361 (+ 98us) write_op.cc:312] APPLY: finished
0902 01:57:55.982829 (+ 468us) log.cc:844] Serialized 40 byte log entry
0902 01:57:55.983114 (+ 285us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:57:55.983358 (+ 244us) write_op.cc:454] Released schema lock
0902 01:57:55.983455 (+ 97us) write_op.cc:341] FINISH: Updating metrics
0902 01:57:56.005404 (+ 21949us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:57:56.005461 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11553,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":78435,"apply.run_cpu_time_us":3600,"apply.run_wall_time_us":27135,"num_ops":4,"prepare.queue_time_us":11681,"prepare.run_cpu_time_us":4268,"prepare.run_wall_time_us":161754,"replication_time_us":582983,"thread_start_us":96225,"threads_started":2}]]}
I20250902 01:57:56.139089 31565 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.145787 31565 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.147974 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.150125 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.151769 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.153319 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.153393 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.155476 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.156417 31610 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.156926 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.158365 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:56.161818 31565 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.162940 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 264) took 1027 ms. Trace:
I20250902 01:57:56.163321 30920 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:57:56.163467 31565 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.166913 31565 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.166858 31509 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.165088 31072 rpcz_store.cc:276] 0902 01:57:55.135159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:57:55.841283 (+706124us) service_pool.cc:225] Handling call
0902 01:57:55.841490 (+ 207us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:57:55.841618 (+ 128us) raft_consensus.cc:1532] Early marking committed up to index 1
0902 01:57:56.071705 (+230087us) log.cc:844] Serialized 14 byte log entry
0902 01:57:56.162342 (+ 90637us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:57:56.162432 (+ 90us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:57:56.162500 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:57:56.162595 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:57:56.162720 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1729,"prepare.queue_time_us":134803,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":93,"thread_start_us":281080,"threads_started":2}
I20250902 01:57:56.167672 31509 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } }
I20250902 01:57:56.169471 31509 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:57:56.178606 31565 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.179214 31565 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.180176 31378 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.180872 30967 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.180742 31378 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.181258 31565 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.181681 31509 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.182786 31378 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.185192 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:57:56.186312 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 95080.6us
I20250902 01:57:56.188043 30892 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.188743 31610 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.190006 30892 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.190749 31610 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.193341 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.195076 30920 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:57:56.192042 30892 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.196410 31610 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.197938 30892 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.198683 31610 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.199867 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.199920 30892 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.200886 31610 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.201862 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.202481 30892 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.203256 31610 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.203799 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.204531 30892 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.205322 31610 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.206578 30892 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.207409 31610 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.208662 31621 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.209111 31621 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.211052 31621 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.211313 31115 tablet_bootstrap.cc:654] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:56.211759 31509 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.231276 31509 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.233378 31509 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.213685 31378 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.214265 30892 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.234195 31378 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.235217 30892 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.235976 31617 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.236456 30892 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.236652 31617 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.236711 31378 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.216307 30967 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.238485 31378 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:56.238878 31378 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:56.239398 31378 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.241448 31601 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.242936 30892 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.244027 30967 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.244799 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:56.218497 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.245306 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.245565 31509 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:56.220479 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.224635 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.246145 31509 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:56.225888 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.246910 30994 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.215374 31565 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:57:56.217662 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:56.247058 31509 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.249861 31576 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.250356 31576 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.244899 31608 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.252238 31576 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.252707 31576 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.253166 31345 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.253289 31612 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.253690 31345 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.255703 31497 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.249655 31565 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.256201 31497 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.257043 31574 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.257462 31565 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.253751 31612 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.257941 31497 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.257669 31574 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.258411 31497 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.259116 31612 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.260167 31574 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.258882 31497 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.253139 31576 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.262346 31497 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.255446 31345 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.262559 31576 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.263208 30994 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:57:56.265058 31617 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.268651 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.271317 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:56.271529 31378 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.272105 30917 raft_consensus.cc:2391] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:57:56.265982 30892 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.268327 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.273216 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.273963 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:56.274318 31617 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:56.274669 31617 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.275017 31617 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.265053 30967 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.276129 31565 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.279938 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.279942 30967 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.281307 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.281764 31565 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.282313 30892 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.283166 30967 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.284479 31115 tablet_bootstrap.cc:492] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:56.284667 31610 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.285220 31115 ts_tablet_manager.cc:1397] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.262s user 0.016s sys 0.003s
I20250902 01:57:56.286243 31565 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.251915 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:56.287727 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.288292 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.289139 31115 raft_consensus.cc:357] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.289745 31115 raft_consensus.cc:383] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:56.288158 31574 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250902 01:57:56.289979 31115 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:56.290678 31574 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:56.290570 31115 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.291082 31574 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.284482 31617 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.292078 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.293375 31617 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.295764 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.287361 30892 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.296944 30997 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:57:56.297363 31576 consensus_queue.cc:1046] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:56.298581 30821 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:56.300503 31576 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:56.300822 31576 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:56.301746 31576 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.299754 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.299969 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.303052 31576 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.306914 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.307363 31601 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.307857 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.308516 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.309585 31378 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.310238 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.311638 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.311914 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.315289 31115 ts_tablet_manager.cc:1428] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250902 01:57:56.315929 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.316030 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.317636 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.319218 31000 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250902 01:57:56.319986 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:56.321003 31574 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.325310 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.325901 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:56.326680 31601 raft_consensus.cc:2762] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250902 01:57:56.327776 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.327888 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.329864 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.327780 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:56.331878 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:56.332466 30922 raft_consensus.cc:2391] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 1: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:57:56.333216 31115 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:56.334194 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.327277 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.336083 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.336267 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.337311 31069 raft_consensus.cc:2391] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 1: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:56.338284 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.339087 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.340668 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.343910 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.345670 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.347039 30824 catalog_manager.cc:5582] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:57:56.350234 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.353066 31574 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.354458 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.356748 31115 tablet_bootstrap.cc:654] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
W20250902 01:57:56.356851 30889 leader_election.cc:343] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:56.357229 30889 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.357916 31617 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250902 01:57:56.358996 30889 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.359794 31617 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.360919 30889 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:56.361691 31617 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:56.362174 31617 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:56.362977 31617 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.367453 31607 consensus_queue.cc:1046] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:56.388350 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:56.388267 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.396667 30992 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.398772 30918 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:57:56.398866 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.400381 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.405030 31115 tablet_bootstrap.cc:492] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:56.405515 31115 ts_tablet_manager.cc:1397] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.018s
I20250902 01:57:56.406878 30889 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.407281 31565 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:56.408015 31608 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:56.408231 31565 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:56.408578 31565 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:56.408452 31608 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.409147 31588 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.409411 31565 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.410346 31608 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.412043 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.402652 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.414762 31589 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:56.415098 31589 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:56.422087 31576 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:56.424141 31001 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.415639 31565 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.413626 31619 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:57:56.427820 31619 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.429420 30993 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.430616 31115 raft_consensus.cc:357] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.431191 31115 raft_consensus.cc:383] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:56.431391 31115 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:56.432008 31115 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.416469 31589 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.433660 31576 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.434274 31576 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.436443 31576 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.437909 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.438813 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.453761 31492 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:56.455727 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.454232 31492 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.458451 31492 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.458779 31115 ts_tablet_manager.cc:1428] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.053s user 0.005s sys 0.001s
I20250902 01:57:56.460853 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.462852 31619 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.464103 31000 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
I20250902 01:57:56.465571 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.465802 31589 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.471418 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.472856 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.474639 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.484534 31115 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:56.487448 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.487699 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.489264 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.497134 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.499327 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.502164 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.504057 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.499698 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.512319 31492 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:56.516265 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.516888 31492 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.512537 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:56.518743 31492 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:56.513020 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.523864 30822 catalog_manager.cc:5582] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 1, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:56.528218 31069 raft_consensus.cc:2366] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2.
I20250902 01:57:56.536463 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.549587 31038 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.550707 31038 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.551882 31612 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.552520 31612 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.556515 31115 tablet_bootstrap.cc:654] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:56.562840 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.564698 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.567585 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.575134 30821 catalog_manager.cc:5582] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:56.577730 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.583927 31497 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.586148 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.589246 30967 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.592432 31497 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.597709 31497 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.600385 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.617606 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.617831 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:56.625763 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:57:56.616744 31608 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.635051 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.637868 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.640662 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:56.641350 31068 raft_consensus.cc:2391] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:56.642160 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:56.642640 30889 consensus_peers.cc:489] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Couldn't send request to peer 975ecd397f5146409baac87e56a302e9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250902 01:57:56.644464 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.649734 31345 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.650234 31345 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.652100 31570 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:56.653626 31570 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.654013 31345 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.655443 31570 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:56.655944 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.656571 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.657032 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:56.659348 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.661993 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.662734 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.663164 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.663839 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.664296 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:56.665283 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:57:56.653136 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 63943.6us
I20250902 01:57:56.667598 30967 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.669497 31115 tablet_bootstrap.cc:492] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:56.671576 31115 ts_tablet_manager.cc:1397] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.187s user 0.024s sys 0.017s
I20250902 01:57:56.676311 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.677027 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.678086 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.678404 31640 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.679246 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:56.679400 31075 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:57:56.684847 31115 raft_consensus.cc:357] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.685461 31115 raft_consensus.cc:383] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:56.685720 31115 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:56.686298 31115 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.691355 30967 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:57:56.693296 31038 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.517s (SENT) [suppressed 4 similar messages]
I20250902 01:57:56.696441 31573 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.697542 30967 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.698196 31573 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.699287 30967 leader_election.cc:400] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2.
I20250902 01:57:56.699559 30967 leader_election.cc:403] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:57:56.700038 31573 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2.
I20250902 01:57:56.701071 30967 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.701714 31573 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.702937 30967 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.703505 31573 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.704828 31115 ts_tablet_manager.cc:1428] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.033s user 0.003s sys 0.002s
I20250902 01:57:56.705644 31115 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Bootstrap starting.
I20250902 01:57:56.711165 31115 tablet_bootstrap.cc:654] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Neither blocks nor log segments found. Creating new log.
I20250902 01:57:56.713143 31038 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.713843 31576 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.714308 31038 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.715135 31576 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.715914 30967 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:57:56.716493 31038 leader_election.cc:343] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:56.716569 31640 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.716954 31038 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.717692 31570 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.717824 30967 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.718542 31640 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.718843 31038 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.719558 31570 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.720091 30967 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.720710 31038 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.721444 31576 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.722018 31038 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.722724 31576 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.723275 31038 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.724359 31570 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.724670 31038 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.725354 31576 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:57:56.725711 31038 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.726387 31576 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.726765 31038 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.727691 31640 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.727893 31038 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.728067 31576 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.728605 31576 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.728735 30967 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.728991 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.729383 31640 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.729621 31576 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.730284 30967 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.731058 31640 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.731719 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.731947 30967 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.732461 31576 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.732579 31640 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.732795 31038 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.733554 31576 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.733662 30967 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.734321 31640 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:56.734948 31038 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.735700 31570 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.736089 31038 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.736814 31570 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.737207 30967 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.737824 31640 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.738195 30967 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.738830 31640 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.739164 30967 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.739754 31640 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.740720 30967 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.741350 31640 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.741890 31115 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: No bootstrap required, opened a new log
I20250902 01:57:56.742396 30967 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.742939 31115 ts_tablet_manager.cc:1397] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.003s
I20250902 01:57:56.743117 31640 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.744915 30967 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.745751 31640 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.745590 31115 raft_consensus.cc:357] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.746313 31115 raft_consensus.cc:383] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250902 01:57:56.746614 31115 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Initialized, Role: FOLLOWER
I20250902 01:57:56.742235 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.746806 30967 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.747241 31570 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.747499 31640 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.748330 31038 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.748960 31570 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.749357 30967 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.750054 31640 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.751031 30967 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.751667 31640 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.751731 31038 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.752562 31570 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.752631 30967 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.752977 30992 raft_consensus.cc:1273] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250902 01:57:56.753335 31640 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.753576 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.754206 31570 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.754367 31589 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:56.755695 31115 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.757833 31115 ts_tablet_manager.cc:1428] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s
I20250902 01:57:56.758208 31570 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:56.759727 31038 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.760473 31570 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.762663 31038 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.763365 31570 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.765260 30967 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.765982 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.766491 31640 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:56.766953 31570 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.768194 31038 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.768832 31576 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.769172 31038 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.769829 31576 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:57:56.770139 31038 leader_election.cc:343] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Illegal state: must be running to vote when last-logged opid is not known
I20250902 01:57:56.770628 31038 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.771308 31576 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250902 01:57:56.777047 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.777288 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.777911 31570 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.785568 31640 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.794067 30967 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.794801 31640 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:56.795641 30967 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:56.796437 31608 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.796885 31640 consensus_queue.cc:1046] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:56.797755 31608 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:57:56.798105 31608 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:57:56.798844 31608 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:56.800032 31608 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:56.809544 31069 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:57:56.811568 31069 raft_consensus.cc:1273] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250902 01:57:56.811964 31069 pending_rounds.cc:85] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 0
I20250902 01:57:56.814194 31069 pending_rounds.cc:107] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250902 01:57:56.814544 31069 raft_consensus.cc:2887] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:57:56.812681 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.817631 31611 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:56.820065 31573 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:56.825629 31038 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.826534 31570 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.811865 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.812662 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.827795 30998 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:56.812961 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.813247 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:56.831146 31038 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.834921 31038 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.836212 31574 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.836259 31570 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.836167 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:56.837757 31574 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:57:56.865574 30917 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:57:56.866983 31640 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:57.052253 31573 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:57.052877 31573 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.055411 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:57.056170 31573 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:57.055716 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:57.057570 30967 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:57.058218 31573 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250902 01:57:57.201738 30994 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 1
W20250902 01:57:57.281328 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:57.577382 31611 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:57.577975 31611 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.580364 31611 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:57.580833 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:57.583034 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:57.584124 30994 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:57:57.585604 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:57.586699 31611 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:57:57.587041 31611 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:57.587352 31611 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:57.594190 31611 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.595559 31611 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:57.596423 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:57.596426 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:57.597087 30994 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:57.601528 30994 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:57:57.602277 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:57.602808 31611 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:57:57.603224 31611 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:57.603878 31611 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.618952 30822 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:57.622493 31576 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:57.624109 31576 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.626829 31576 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:57.627142 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:57.627933 31001 raft_consensus.cc:2366] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:57:57.629189 31038 leader_election.cc:400] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:57:57.629527 31038 leader_election.cc:403] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:57:57.630510 31574 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:57:57.638207 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:57.639839 31570 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:57.640425 31570 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.642561 31570 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:57.639474 31574 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:57:57.644160 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:57.645567 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:57.646116 31001 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:57.647250 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:57.648084 31574 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:57.648396 31574 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:57.648767 31574 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:57.653856 31574 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.655471 31574 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:57.655814 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:57.656033 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:57.659921 31001 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:57.660647 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:57.661231 31574 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:57.661710 31574 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:57.662556 31574 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:57.676142 30822 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:58.073725 31588 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:58.074115 31588 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.075635 31588 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:58.076437 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:58.076635 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:58.077123 31075 raft_consensus.cc:2391] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:58.077150 30994 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:57:58.078202 30889 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:58.078895 31588 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:58.079174 31588 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:57:58.079422 31588 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:58.083352 30994 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:58.084582 31588 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.084558 31570 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:57:58.086195 31588 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:58.086953 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:58.087416 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:57:58.088121 31075 raft_consensus.cc:2391] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:57:58.105495 30994 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:57:58.106915 30889 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:57:58.109385 31655 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:58.119038 31655 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:58.122220 31655 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.146543 30923 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:58.146646 30994 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:57:58.150357 30822 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:57:58.150852 31659 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:58.157500 31662 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:58.162510 31069 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:57:58.168624 31662 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250902 01:57:58.311486 30872 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250902 01:57:58.551676 31069 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:58.552852 31659 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:58.552953 31662 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:58.553321 31659 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.555250 31659 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:58.555680 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:58.556700 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:58.557257 30997 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:58.557947 31038 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:57:58.558571 31576 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:57:58.558915 31576 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:57:58.559229 31576 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:58.569085 30994 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:57:58.573944 31576 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.576175 31069 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:57:58.578130 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:58.579689 31660 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:58.585732 31662 consensus_queue.cc:1046] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:57:58.589035 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:58.589150 31576 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:58.592689 31655 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:57:58.593032 31655 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:57:58.593735 31655 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.594784 31655 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:57:58.595857 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:58.596799 31570 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:57:58.931262 31570 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:58.931794 31570 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.933818 31570 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:58.934885 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:58.935160 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:57:58.935472 30994 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 0.
I20250902 01:57:58.935743 30923 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:57:58.936486 31038 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:57:58.937179 31570 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:57:58.937526 31570 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:57:58.937837 31570 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:58.944265 31570 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.946856 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:57:58.947070 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:57:58.947655 31570 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:57:58.947412 30994 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 0 FOLLOWER]: Advancing to term 2
I20250902 01:57:58.947691 30920 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:57:58.959654 30920 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:58.960472 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:57:58.961252 31570 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:57:58.961651 31570 raft_consensus.cc:695] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:57:58.961583 30994 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:57:58.962353 31570 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:58.972958 30821 catalog_manager.cc:5582] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 0 to 2, leader changed from <none> to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:57:59.419273 30923 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250902 01:57:59.420212 31576 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250902 01:57:59.430701 30994 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250902 01:57:59.432060 31678 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250902 01:57:59.806664 29783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250902 01:57:59.846999 31683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:59.850495 31684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250902 01:57:59.856032 31686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250902 01:57:59.862982 29783 server_base.cc:1047] running on GCE node
I20250902 01:57:59.865072 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250902 01:57:59.865368 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250902 01:57:59.865562 29783 hybrid_clock.cc:648] HybridClock initialized: now 1756778279865545 us; error 0 us; skew 500 ppm
I20250902 01:57:59.866223 29783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250902 01:57:59.869412 29783 webserver.cc:480] Webserver started at http://127.29.21.196:43917/ using document root <none> and password file <none>
I20250902 01:57:59.869850 29783 fs_manager.cc:362] Metadata directory not provided
I20250902 01:57:59.870033 29783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250902 01:57:59.870297 29783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250902 01:57:59.871832 29783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-3-root/instance:
uuid: "01f6731ce5044a6a948a5ae04da80e6f"
format_stamp: "Formatted at 2025-09-02 01:57:59 on dist-test-slave-g0c9"
I20250902 01:57:59.877385 29783 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250902 01:57:59.882835 31691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:59.883692 29783 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250902 01:57:59.884011 29783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-3-root
uuid: "01f6731ce5044a6a948a5ae04da80e6f"
format_stamp: "Formatted at 2025-09-02 01:57:59 on dist-test-slave-g0c9"
I20250902 01:57:59.884274 29783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250902 01:57:59.902655 31682 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:57:59.903223 31682 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:57:59.905253 29783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250902 01:57:59.905345 31682 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:57:59.905915 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:57:59.906980 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:57:59.907230 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250902 01:57:59.908360 30892 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:57:59.909049 29783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250902 01:57:59.909415 31682 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:57:59.912263 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250902 01:57:59.912488 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250902 01:57:59.912703 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250902 01:57:59.912845 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250902 01:58:00.053575 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.196:44285
I20250902 01:58:00.054106 31754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.21.196:44285 every 8 connection(s)
I20250902 01:58:00.067831 29783 auto_leader_rebalancer-test.cc:417] add a tserver: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:00.083292 31573 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:00.083789 31573 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.085513 31573 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:00.086804 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:00.086795 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:00.087513 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:00.087587 31069 raft_consensus.cc:2391] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:00.089922 30965 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:00.090536 31573 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:00.090991 31573 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:00.091380 31573 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:00.099161 31755 heartbeater.cc:344] Connected to a master server at 127.29.21.254:45755
I20250902 01:58:00.099581 31755 heartbeater.cc:461] Registering TS with master...
I20250902 01:58:00.100481 31755 heartbeater.cc:507] Master 127.29.21.254:45755 requested a full tablet report, sending...
I20250902 01:58:00.102196 31573 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.103402 30821 ts_manager.cc:194] Registered new tserver with Master: 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:00.105177 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:00.105291 30821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40172
I20250902 01:58:00.106089 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:00.106801 31070 raft_consensus.cc:2391] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:00.108664 31573 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:00.111413 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:00.112352 30965 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:00.112958 31573 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:00.113488 31573 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:00.114334 31573 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.129585 30821 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 3, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:00.445389 31069 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:00.457433 30922 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.458626 31678 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.473096 31678 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.476739 30920 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.488101 31001 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.493935 31767 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.495121 30821 catalog_manager.cc:5582] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:00.520711 31001 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.525913 30922 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:00.536307 31069 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.539175 31655 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.545068 31768 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:00.545840 31768 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.548573 31768 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:00.549715 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:00.560516 31682 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.569599 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:00.570247 31001 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.570989 31069 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.570187 30918 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
W20250902 01:58:00.580945 31038 consensus_peers.cc:489] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f160859afd5945949c030026bfe3ac61. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:00.582321 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:00.583905 31709 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:00.584254 31709 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:00.584554 31709 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:00.591370 31709 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.593621 31662 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.595860 31709 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:00.596438 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:00.596995 30922 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:00.603729 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:00.603832 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:00.606935 30821 catalog_manager.cc:5582] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.609876 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:00.613961 31709 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:00.614590 31709 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:00.615384 31709 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.624460 31069 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:00.628557 30889 consensus_peers.cc:489] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 331364bf121c480483b11ff32993e1ec. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:00.655197 30821 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:00.656852 31001 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.674945 30994 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:00.675571 31038 consensus_peers.cc:489] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 744f11902df74dea8914fdd4a992259d. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:00.676223 30924 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.677124 31678 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.678584 31768 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.690371 31768 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.692600 30994 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.692744 30924 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.708709 30821 catalog_manager.cc:5582] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:00.713619 31069 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:00.728528 31038 consensus_peers.cc:489] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26ad5d89b038487b8bbc776913d5a215. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:00.731062 30994 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.731303 30924 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:00.732617 31709 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.733541 31678 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:00.739454 30922 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:00.741394 31573 consensus_queue.cc:1046] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:00.742333 31656 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:00.742786 31656 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:00.743604 31656 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:00.744866 31656 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:00.749480 31678 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.754683 31001 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.756444 31069 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:00.771742 30922 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:00.773099 30821 catalog_manager.cc:5582] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:01.050568 31789 ts_tablet_manager.cc:927] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:01.054073 31790 ts_tablet_manager.cc:927] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:01.057104 31789 tablet_copy_client.cc:323] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:01.058954 31790 tablet_copy_client.cc:323] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:01.108212 31755 heartbeater.cc:499] Master 127.29.21.254:45755 was elected leader, sending a full tablet report...
I20250902 01:58:01.112202 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 331364bf121c480483b11ff32993e1ec from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:01.112763 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 331364bf121c480483b11ff32993e1ec from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-331364bf121c480483b11ff32993e1ec
I20250902 01:58:01.114022 31085 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet f160859afd5945949c030026bfe3ac61 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:01.114614 31085 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet f160859afd5945949c030026bfe3ac61 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-f160859afd5945949c030026bfe3ac61
I20250902 01:58:01.122048 30936 tablet_copy_source_session.cc:215] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:01.126092 31790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 331364bf121c480483b11ff32993e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:01.133074 31085 tablet_copy_source_session.cc:215] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:01.139351 31789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f160859afd5945949c030026bfe3ac61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:01.149729 31797 ts_tablet_manager.cc:927] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:01.151254 30994 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250902 01:58:01.151693 30994 pending_rounds.cc:85] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 2
I20250902 01:58:01.151969 30994 pending_rounds.cc:107] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250902 01:58:01.152302 30994 raft_consensus.cc:2887] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:01.153890 31678 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:01.162832 31789 tablet_copy_client.cc:806] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:01.163473 31789 tablet_copy_client.cc:670] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:01.184306 31789 tablet_copy_client.cc:538] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:01.205626 30924 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:01.206928 31678 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:01.177656 31790 tablet_copy_client.cc:806] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:01.218155 31790 tablet_copy_client.cc:670] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:01.239262 31790 tablet_copy_client.cc:538] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:01.284870 31789 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:01.285818 31802 ts_tablet_manager.cc:927] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:01.290230 31790 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:01.290400 31797 tablet_copy_client.cc:323] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:01.303894 31085 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 744f11902df74dea8914fdd4a992259d from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:01.304383 31085 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 744f11902df74dea8914fdd4a992259d from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-744f11902df74dea8914fdd4a992259d
I20250902 01:58:01.313756 31085 tablet_copy_source_session.cc:215] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:01.322549 31797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 744f11902df74dea8914fdd4a992259d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:01.335107 31802 tablet_copy_client.cc:323] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:01.341724 31085 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 26ad5d89b038487b8bbc776913d5a215 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:01.342307 31085 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 26ad5d89b038487b8bbc776913d5a215 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-26ad5d89b038487b8bbc776913d5a215
I20250902 01:58:01.353567 31797 tablet_copy_client.cc:806] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:01.354251 31797 tablet_copy_client.cc:670] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:01.356774 31085 tablet_copy_source_session.cc:215] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:01.368224 31802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ad5d89b038487b8bbc776913d5a215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:01.368997 31797 tablet_copy_client.cc:538] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:01.395031 31789 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:01.398046 31797 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:01.410037 31790 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:01.412801 31790 tablet_bootstrap.cc:492] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:01.413481 31790 ts_tablet_manager.cc:1397] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.124s user 0.049s sys 0.019s
I20250902 01:58:01.412004 31789 tablet_bootstrap.cc:492] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:01.417492 31789 ts_tablet_manager.cc:1397] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.133s user 0.021s sys 0.037s
I20250902 01:58:01.418278 31790 raft_consensus.cc:357] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.420215 31802 tablet_copy_client.cc:806] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:01.420244 31790 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:01.420766 31802 tablet_copy_client.cc:670] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:01.422497 31790 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.424104 31802 tablet_copy_client.cc:538] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:01.424703 31789 raft_consensus.cc:357] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.428378 31789 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:01.429052 31789 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.435361 31802 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:01.452899 31790 ts_tablet_manager.cc:1428] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.039s user 0.008s sys 0.006s
I20250902 01:58:01.459015 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-331364bf121c480483b11ff32993e1ec received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:01.459646 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-331364bf121c480483b11ff32993e1ec on tablet 331364bf121c480483b11ff32993e1ec with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:01.472517 31797 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:01.473397 31797 tablet_bootstrap.cc:492] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:01.474020 31797 ts_tablet_manager.cc:1397] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.012s
I20250902 01:58:01.479439 31797 raft_consensus.cc:357] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.480301 31797 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:01.480973 31797 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.481998 31729 raft_consensus.cc:1215] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:01.483356 31797 ts_tablet_manager.cc:1428] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250902 01:58:01.484786 31789 ts_tablet_manager.cc:1428] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.067s user 0.018s sys 0.009s
I20250902 01:58:01.495834 31085 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-744f11902df74dea8914fdd4a992259d received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:01.496392 31085 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-744f11902df74dea8914fdd4a992259d on tablet 744f11902df74dea8914fdd4a992259d with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:01.504168 31085 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-f160859afd5945949c030026bfe3ac61 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:01.504690 31085 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-f160859afd5945949c030026bfe3ac61 on tablet f160859afd5945949c030026bfe3ac61 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:01.534152 31802 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:01.535156 31802 tablet_bootstrap.cc:492] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:01.536260 31802 ts_tablet_manager.cc:1397] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.101s user 0.021s sys 0.024s
I20250902 01:58:01.539866 31802 raft_consensus.cc:357] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.540676 31802 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:01.542075 31802 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:01.544708 31802 ts_tablet_manager.cc:1428] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250902 01:58:01.550355 31085 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-26ad5d89b038487b8bbc776913d5a215 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:01.550733 31085 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-26ad5d89b038487b8bbc776913d5a215 on tablet 26ad5d89b038487b8bbc776913d5a215 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:01.558606 31729 raft_consensus.cc:1215] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:01.602301 31729 raft_consensus.cc:1215] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:01.810966 31729 raft_consensus.cc:1215] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:01.854296 31768 raft_consensus.cc:1062] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:01.856427 31768 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:01.862422 30994 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:01.864511 31729 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:01.864883 30922 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:01.865927 31709 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:01.868207 31767 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:01.870250 31768 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:01.881942 31709 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:01.887029 31001 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:01.888878 31730 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:01.901486 30824 catalog_manager.cc:5582] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:01.919921 30922 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:01.960609 31767 raft_consensus.cc:1062] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:01.962749 31767 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:01.969156 31070 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:01.979986 30994 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:01.980170 31730 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:01.985370 31709 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:01.987637 31768 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.028406 30922 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.032172 31768 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.033048 30994 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.033056 31729 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.042090 31768 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.044183 31822 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.043898 31768 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.054479 31730 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.057520 31001 raft_consensus.cc:2953] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.075884 31709 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.078296 30994 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.087980 31780 raft_consensus.cc:1062] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:02.089758 31780 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:02.093127 30922 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.095171 30821 catalog_manager.cc:5582] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.114894 30823 catalog_manager.cc:5582] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.127882 30994 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.127911 31075 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.130436 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet f160859afd5945949c030026bfe3ac61 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:02.131515 31662 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.131881 31730 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.133179 31662 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250902 01:58:02.137643 31682 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.164949 31827 tablet_replica.cc:331] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:02.166316 31827 raft_consensus.cc:2241] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:02.170694 31827 raft_consensus.cc:2270] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:02.198205 31729 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.205688 31827 ts_tablet_manager.cc:1905] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:02.208268 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f160859afd5945949c030026bfe3ac61 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:02.235544 31069 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:02.275792 31780 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:02.280184 31830 log.cc:927] Time spent T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20250902 01:58:02.282296 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 100236us
I20250902 01:58:02.284157 30922 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.291074 30994 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.288069 31709 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20250902 01:58:02.309330 31730 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.311868 31822 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.312227 30823 catalog_manager.cc:5582] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:02.313225 31827 ts_tablet_manager.cc:1918] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:02.313730 31827 log.cc:1199] T f160859afd5945949c030026bfe3ac61 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/f160859afd5945949c030026bfe3ac61
W20250902 01:58:02.332170 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49878.2us
I20250902 01:58:02.340297 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet f160859afd5945949c030026bfe3ac61 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:02.358726 31730 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.360083 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 744f11902df74dea8914fdd4a992259d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:02.335393 31075 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.361793 30924 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:02.369264 30922 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.372617 30821 catalog_manager.cc:5582] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:02.401291 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 744f11902df74dea8914fdd4a992259d with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:02.401139 31822 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.402068 31730 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.403012 30994 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.403414 31834 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.404381 31767 raft_consensus.cc:1062] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:02.406983 31767 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:02.409881 31834 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.412501 31780 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.415217 31730 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.423646 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 331364bf121c480483b11ff32993e1ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250902 01:58:02.494274 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 95643.2us
I20250902 01:58:02.489003 31729 raft_consensus.cc:2953] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.502863 30821 catalog_manager.cc:5582] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:02.538964 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 331364bf121c480483b11ff32993e1ec with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:02.549429 31729 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.554642 31768 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.556185 31001 raft_consensus.cc:2953] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.557041 31837 tablet_replica.cc:331] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:02.564992 30922 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.565268 31840 tablet_replica.cc:331] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:02.566219 31768 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.565178 31837 raft_consensus.cc:2241] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:02.586711 31837 raft_consensus.cc:2270] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:02.589958 31837 ts_tablet_manager.cc:1905] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:02.590857 31840 raft_consensus.cc:2241] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:02.591678 31840 raft_consensus.cc:2270] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:02.597175 31840 ts_tablet_manager.cc:1905] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:02.601541 31001 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:02.609750 31768 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:02.627197 31835 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.630270 31840 ts_tablet_manager.cc:1918] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:02.630796 31840 log.cc:1199] T 331364bf121c480483b11ff32993e1ec P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/331364bf121c480483b11ff32993e1ec
I20250902 01:58:02.634078 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 331364bf121c480483b11ff32993e1ec (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:02.636245 30924 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.646312 31729 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.652130 30821 catalog_manager.cc:5582] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:02.661356 31837 ts_tablet_manager.cc:1918] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:02.666407 31837 log.cc:1199] T 744f11902df74dea8914fdd4a992259d P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/744f11902df74dea8914fdd4a992259d
I20250902 01:58:02.668087 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 744f11902df74dea8914fdd4a992259d (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:02.670733 31001 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.676566 31072 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:02.700323 30924 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.701537 31729 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:02.703194 31709 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.704438 31835 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:02.709136 31709 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.711225 30924 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.719383 31729 raft_consensus.cc:2953] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.723701 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 26ad5d89b038487b8bbc776913d5a215 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:02.733064 30821 catalog_manager.cc:5582] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:02.741441 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 26ad5d89b038487b8bbc776913d5a215 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:02.742393 31837 tablet_replica.cc:331] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:02.743361 31837 raft_consensus.cc:2241] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:02.744032 31837 raft_consensus.cc:2270] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:02.748200 31837 ts_tablet_manager.cc:1905] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:02.765142 31837 ts_tablet_manager.cc:1918] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:02.765547 31837 log.cc:1199] T 26ad5d89b038487b8bbc776913d5a215 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/26ad5d89b038487b8bbc776913d5a215
I20250902 01:58:02.769174 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 26ad5d89b038487b8bbc776913d5a215 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:03.802644 31001 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:03.807682 31072 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.807791 30924 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.808847 31656 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:03.809473 31848 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250902 01:58:03.824505 30964 consensus_peers.cc:489] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7f66bcc25964883baf67f48889ec766. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:03.829548 31656 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.832365 31072 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.839694 31070 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:03.831367 30924 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.844450 30821 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:03.863152 31038 consensus_peers.cc:489] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f5d81000607410da8880ef8e5517680. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:03.864704 30994 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.866667 30924 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.866616 31835 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:03.868250 31835 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:03.878417 31835 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.886399 30924 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.905234 31001 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.914490 31072 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:03.915331 30821 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
W20250902 01:58:03.920397 31038 consensus_peers.cc:489] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63fa3c697d3e40afb9a01578cbe72700. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:03.920464 31001 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.920845 30922 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.922169 31860 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:03.927202 31768 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:03.940867 31835 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.947531 30924 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.949512 31001 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:03.953738 31072 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:03.961117 30821 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:03.992937 31038 consensus_peers.cc:489] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8ce2ebbb1c24a6c998649ed3656707a. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:03.997263 30994 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:03.998891 31835 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:04.016125 30923 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:04.017149 31835 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:04.024822 31768 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:04.026533 31001 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:04.028079 30918 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:04.043097 30821 catalog_manager.cc:5582] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:04.295814 31871 ts_tablet_manager.cc:927] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:04.301719 31871 tablet_copy_client.cc:323] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:04.311031 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet d7f66bcc25964883baf67f48889ec766 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:04.311515 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet d7f66bcc25964883baf67f48889ec766 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-d7f66bcc25964883baf67f48889ec766
I20250902 01:58:04.318450 31011 tablet_copy_source_session.cc:215] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:04.322078 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f66bcc25964883baf67f48889ec766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:04.333388 31871 tablet_copy_client.cc:806] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:04.333845 31871 tablet_copy_client.cc:670] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:04.337162 31871 tablet_copy_client.cc:538] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:04.345263 31871 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:04.368916 31871 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:04.369580 31871 tablet_bootstrap.cc:492] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:04.370201 31871 ts_tablet_manager.cc:1397] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.001s
I20250902 01:58:04.372677 31871 raft_consensus.cc:357] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.373251 31871 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:04.373710 31871 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.376705 31871 ts_tablet_manager.cc:1428] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250902 01:58:04.378515 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d7f66bcc25964883baf67f48889ec766 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:04.378930 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d7f66bcc25964883baf67f48889ec766 on tablet d7f66bcc25964883baf67f48889ec766 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:04.434870 31871 ts_tablet_manager.cc:927] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:04.436734 31871 tablet_copy_client.cc:323] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:04.438391 31085 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 63fa3c697d3e40afb9a01578cbe72700 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:04.438844 31085 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 63fa3c697d3e40afb9a01578cbe72700 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-63fa3c697d3e40afb9a01578cbe72700
I20250902 01:58:04.446149 31085 tablet_copy_source_session.cc:215] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:04.449090 31871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fa3c697d3e40afb9a01578cbe72700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:04.463379 31871 tablet_copy_client.cc:806] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:04.463833 31871 tablet_copy_client.cc:670] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:04.468833 31878 ts_tablet_manager.cc:927] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:04.471136 31878 tablet_copy_client.cc:323] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:04.475580 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 8f5d81000607410da8880ef8e5517680 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:04.476233 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 8f5d81000607410da8880ef8e5517680 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-8f5d81000607410da8880ef8e5517680
I20250902 01:58:04.494855 31084 tablet_copy_source_session.cc:215] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:04.496117 31871 tablet_copy_client.cc:538] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:04.497874 31878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f5d81000607410da8880ef8e5517680. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:04.514966 31878 tablet_copy_client.cc:806] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:04.515573 31878 tablet_copy_client.cc:670] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:04.516254 31871 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:04.519876 31878 tablet_copy_client.cc:538] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:04.530181 31878 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:04.531971 31881 ts_tablet_manager.cc:927] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:04.536123 31871 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:04.536979 31871 tablet_bootstrap.cc:492] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:04.537616 31871 ts_tablet_manager.cc:1397] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20250902 01:58:04.541796 31871 raft_consensus.cc:357] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.542850 31871 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:04.544240 31871 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.544809 31881 tablet_copy_client.cc:323] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:04.546726 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet d8ce2ebbb1c24a6c998649ed3656707a from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:04.547176 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet d8ce2ebbb1c24a6c998649ed3656707a from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-d8ce2ebbb1c24a6c998649ed3656707a
I20250902 01:58:04.560398 31084 tablet_copy_source_session.cc:215] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:04.562196 31871 ts_tablet_manager.cc:1428] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20250902 01:58:04.563766 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-63fa3c697d3e40afb9a01578cbe72700 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:04.564149 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-63fa3c697d3e40afb9a01578cbe72700 on tablet 63fa3c697d3e40afb9a01578cbe72700 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:04.564749 31881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8ce2ebbb1c24a6c998649ed3656707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:04.572512 31878 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:04.574036 31878 tablet_bootstrap.cc:492] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:04.574604 31878 ts_tablet_manager.cc:1397] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.004s
I20250902 01:58:04.576977 31878 raft_consensus.cc:357] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.577710 31878 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:04.578269 31878 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.580368 31878 ts_tablet_manager.cc:1428] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250902 01:58:04.582653 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-8f5d81000607410da8880ef8e5517680 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:04.583063 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-8f5d81000607410da8880ef8e5517680 on tablet 8f5d81000607410da8880ef8e5517680 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:04.585364 31881 tablet_copy_client.cc:806] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:04.586104 31881 tablet_copy_client.cc:670] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:04.589529 31881 tablet_copy_client.cc:538] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:04.597860 31881 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:04.617555 31881 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:04.618539 31881 tablet_bootstrap.cc:492] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:04.619123 31881 ts_tablet_manager.cc:1397] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20250902 01:58:04.621677 31881 raft_consensus.cc:357] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.622423 31881 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:04.623034 31881 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:04.625128 31881 ts_tablet_manager.cc:1428] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20250902 01:58:04.627796 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d8ce2ebbb1c24a6c998649ed3656707a received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:04.628152 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d8ce2ebbb1c24a6c998649ed3656707a on tablet d8ce2ebbb1c24a6c998649ed3656707a with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:04.807363 31729 raft_consensus.cc:1215] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:04.853628 31729 raft_consensus.cc:1215] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:04.999397 31729 raft_consensus.cc:1215] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:05.046419 31729 raft_consensus.cc:1215] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:05.233047 31656 raft_consensus.cc:1062] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:05.234848 31656 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.248869 30923 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.248795 31729 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.248212 31072 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.250352 31848 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:05.251571 31853 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.253425 31887 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.261300 31887 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.262823 31729 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.264586 31069 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.274503 30923 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.280084 30824 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.309130 30994 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.314496 30923 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.314522 31729 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.315796 31853 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.316532 31656 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.322976 31853 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.330080 31729 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.330080 30918 raft_consensus.cc:2953] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.333475 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d7f66bcc25964883baf67f48889ec766 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:05.342118 30821 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:05.357067 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet d7f66bcc25964883baf67f48889ec766 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:05.358770 31895 tablet_replica.cc:331] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:05.359665 31895 raft_consensus.cc:2241] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:05.360312 31895 raft_consensus.cc:2270] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:05.366951 31895 ts_tablet_manager.cc:1905] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:05.375787 31860 raft_consensus.cc:1062] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:05.377751 31860 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.380472 31895 ts_tablet_manager.cc:1918] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:05.380834 31895 log.cc:1199] T d7f66bcc25964883baf67f48889ec766 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/d7f66bcc25964883baf67f48889ec766
I20250902 01:58:05.382345 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet d7f66bcc25964883baf67f48889ec766 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:05.387641 30994 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.388813 31729 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.389014 31861 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.391016 30918 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.392242 31768 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.393014 31861 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.409761 31861 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.411662 30918 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.412447 31730 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.430043 30994 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.433015 30822 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:05.447165 31069 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.452543 30994 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.454526 31730 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.455979 31768 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.456548 31861 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.463480 31897 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.466439 31001 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.470933 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 63fa3c697d3e40afb9a01578cbe72700 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:05.474932 31730 raft_consensus.cc:2953] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.485252 30822 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:05.500386 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 63fa3c697d3e40afb9a01578cbe72700 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:05.502586 31902 tablet_replica.cc:331] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:05.503578 31902 raft_consensus.cc:2241] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:05.504225 31902 raft_consensus.cc:2270] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:05.506947 31902 ts_tablet_manager.cc:1905] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:05.519181 31902 ts_tablet_manager.cc:1918] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:05.519487 31902 log.cc:1199] T 63fa3c697d3e40afb9a01578cbe72700 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/63fa3c697d3e40afb9a01578cbe72700
I20250902 01:58:05.521100 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 63fa3c697d3e40afb9a01578cbe72700 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:05.618974 31897 raft_consensus.cc:1062] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:05.620919 31897 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.626075 31001 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.626459 31730 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.627151 30918 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.627255 31897 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.628150 31861 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.628916 31901 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.641026 31901 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.642918 30918 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.642592 31730 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.655319 30822 catalog_manager.cc:5582] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.657610 31001 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.663125 31897 raft_consensus.cc:1062] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:05.668041 31897 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.679893 31070 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.688450 31730 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.690779 31001 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.691030 30918 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:05.692096 31897 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.696417 31861 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.697760 31768 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:05.700590 31730 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.701678 30918 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.703137 31861 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.703953 31860 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.712260 31860 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.713249 31768 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.714599 31001 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.721074 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d8ce2ebbb1c24a6c998649ed3656707a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:05.722427 30918 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.727587 31730 raft_consensus.cc:2953] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.740157 30822 catalog_manager.cc:5582] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:05.740131 30918 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.746213 30821 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.772855 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet d8ce2ebbb1c24a6c998649ed3656707a with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:05.773698 31729 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.790822 31911 tablet_replica.cc:331] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:05.791965 31911 raft_consensus.cc:2241] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:05.793073 31911 raft_consensus.cc:2270] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:05.798563 31911 ts_tablet_manager.cc:1905] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:05.818935 31911 ts_tablet_manager.cc:1918] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:05.819298 31911 log.cc:1199] T d8ce2ebbb1c24a6c998649ed3656707a P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/d8ce2ebbb1c24a6c998649ed3656707a
I20250902 01:58:05.820564 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet d8ce2ebbb1c24a6c998649ed3656707a (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:05.820994 31072 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:05.825536 31729 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.825845 31001 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:05.827466 31897 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.829118 31861 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:05.834215 31897 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.838299 31729 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.840994 31001 raft_consensus.cc:2953] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.848349 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8f5d81000607410da8880ef8e5517680 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:05.850698 30822 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:05.861366 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 8f5d81000607410da8880ef8e5517680 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:05.862273 31902 tablet_replica.cc:331] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:05.863231 31902 raft_consensus.cc:2241] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:05.863955 31902 raft_consensus.cc:2270] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:05.866672 31902 ts_tablet_manager.cc:1905] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:05.875851 31902 ts_tablet_manager.cc:1918] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:05.876215 31902 log.cc:1199] T 8f5d81000607410da8880ef8e5517680 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/8f5d81000607410da8880ef8e5517680
I20250902 01:58:05.877332 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 8f5d81000607410da8880ef8e5517680 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:06.940894 30994 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:06.946076 31069 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:06.947635 30964 consensus_peers.cc:489] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b03cb35549a346d28baa21d708d665e2. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:06.947587 31915 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:06.948271 30923 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:06.949828 31853 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:06.955859 31853 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:06.957324 31069 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:06.961730 30923 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:06.970111 31072 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:06.976853 30823 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:06.996721 31038 consensus_peers.cc:489] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ede06f18c6574c57857475151e3767d2. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:06.997215 31001 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:58:06.998406 31897 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:07.002879 30923 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:58:07.003976 31897 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:07.021749 31901 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.024227 31001 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.032131 30923 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.040293 31072 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.046757 30824 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:07.052986 30994 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:07.053390 31038 consensus_peers.cc:489] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0692c5029b5e4ab088324d038ff964be. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:07.054605 30923 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:07.054584 31923 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:07.055953 31901 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:07.062130 31923 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.063439 30994 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.065407 30923 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.079807 30924 consensus_queue.cc:237] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.088130 30822 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.093634 31072 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:07.094657 30889 consensus_peers.cc:489] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d30dc90c304472ab905d3af8400b8d0. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:07.095412 31913 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:07.097983 30994 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:07.100222 31883 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:07.104925 31919 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.106254 31072 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.110659 31001 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.122078 30823 catalog_manager.cc:5582] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:07.542244 31933 ts_tablet_manager.cc:927] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:07.543949 31933 tablet_copy_client.cc:323] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:07.545435 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet b03cb35549a346d28baa21d708d665e2 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:07.545931 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet b03cb35549a346d28baa21d708d665e2 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-b03cb35549a346d28baa21d708d665e2
I20250902 01:58:07.551668 31011 tablet_copy_source_session.cc:215] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:07.554564 31933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03cb35549a346d28baa21d708d665e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:07.567050 31933 tablet_copy_client.cc:806] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:07.567657 31933 tablet_copy_client.cc:670] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:07.571178 31933 tablet_copy_client.cc:538] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:07.574404 31935 ts_tablet_manager.cc:927] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:07.577189 31935 tablet_copy_client.cc:323] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:07.578505 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet ede06f18c6574c57857475151e3767d2 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:07.578943 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet ede06f18c6574c57857475151e3767d2 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-ede06f18c6574c57857475151e3767d2
I20250902 01:58:07.580556 31933 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:07.591356 31084 tablet_copy_source_session.cc:215] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:07.594169 31935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede06f18c6574c57857475151e3767d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:07.600998 31933 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:07.601859 31933 tablet_bootstrap.cc:492] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:07.602491 31933 ts_tablet_manager.cc:1397] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.006s
I20250902 01:58:07.604988 31933 raft_consensus.cc:357] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.605716 31933 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:07.606294 31933 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.608186 31935 tablet_copy_client.cc:806] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:07.608673 31935 tablet_copy_client.cc:670] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:07.611230 31933 ts_tablet_manager.cc:1428] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
I20250902 01:58:07.612706 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-b03cb35549a346d28baa21d708d665e2 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:07.613076 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-b03cb35549a346d28baa21d708d665e2 on tablet b03cb35549a346d28baa21d708d665e2 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:07.616850 31939 ts_tablet_manager.cc:927] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:07.617044 31935 tablet_copy_client.cc:538] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:07.625980 31935 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:07.636634 31939 tablet_copy_client.cc:323] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:07.639716 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 0692c5029b5e4ab088324d038ff964be from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:07.640161 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 0692c5029b5e4ab088324d038ff964be from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-0692c5029b5e4ab088324d038ff964be
I20250902 01:58:07.645696 31084 tablet_copy_source_session.cc:215] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:07.648331 31939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0692c5029b5e4ab088324d038ff964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:07.660512 31939 tablet_copy_client.cc:806] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:07.661003 31939 tablet_copy_client.cc:670] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:07.662082 31935 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:07.663102 31935 tablet_bootstrap.cc:492] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:07.664837 31939 tablet_copy_client.cc:538] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:07.664863 31935 ts_tablet_manager.cc:1397] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.008s
I20250902 01:58:07.669607 31935 raft_consensus.cc:357] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.670320 31935 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:07.670919 31935 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.674311 31939 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:07.674741 31935 ts_tablet_manager.cc:1428] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250902 01:58:07.676157 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-ede06f18c6574c57857475151e3767d2 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:07.676554 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-ede06f18c6574c57857475151e3767d2 on tablet ede06f18c6574c57857475151e3767d2 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:07.702811 31935 ts_tablet_manager.cc:927] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:07.753911 31935 tablet_copy_client.cc:323] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:07.755690 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 6d30dc90c304472ab905d3af8400b8d0 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:07.756191 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 6d30dc90c304472ab905d3af8400b8d0 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-6d30dc90c304472ab905d3af8400b8d0
I20250902 01:58:07.763607 30936 tablet_copy_source_session.cc:215] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:07.765169 31939 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:07.766073 31939 tablet_bootstrap.cc:492] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:07.766724 31939 ts_tablet_manager.cc:1397] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.093s user 0.029s sys 0.048s
I20250902 01:58:07.767495 31935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d30dc90c304472ab905d3af8400b8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:07.769012 31939 raft_consensus.cc:357] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.769857 31939 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:07.770480 31939 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.773147 31939 ts_tablet_manager.cc:1428] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:58:07.775228 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-0692c5029b5e4ab088324d038ff964be received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:07.775652 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-0692c5029b5e4ab088324d038ff964be on tablet 0692c5029b5e4ab088324d038ff964be with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:07.783564 31935 tablet_copy_client.cc:806] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:07.784054 31935 tablet_copy_client.cc:670] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:07.787153 31935 tablet_copy_client.cc:538] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:07.793994 31935 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:07.813545 31935 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:07.814333 31935 tablet_bootstrap.cc:492] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:07.814873 31935 ts_tablet_manager.cc:1397] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.014s
I20250902 01:58:07.817540 31935 raft_consensus.cc:357] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.818331 31935 raft_consensus.cc:738] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:07.818969 31935 consensus_queue.cc:260] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:07.821569 31935 ts_tablet_manager.cc:1428] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:58:07.823014 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-6d30dc90c304472ab905d3af8400b8d0 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:07.823380 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-6d30dc90c304472ab905d3af8400b8d0 on tablet 6d30dc90c304472ab905d3af8400b8d0 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:07.963658 31729 raft_consensus.cc:1215] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20250902 01:58:08.046983 31729 raft_consensus.cc:1215] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:08.209028 31729 raft_consensus.cc:1215] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:08.268918 31729 raft_consensus.cc:1215] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:08.396699 31941 raft_consensus.cc:1062] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:08.402923 31941 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.419037 31729 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250902 01:58:08.419607 30997 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250902 01:58:08.420466 30923 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250902 01:58:08.420782 31860 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:08.421501 31941 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:08.422402 31901 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:08.445863 31946 raft_consensus.cc:1062] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:08.448540 31946 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.449733 31901 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.455404 30997 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.461957 31729 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.468510 31730 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.469568 30924 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.469568 31072 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.470268 31887 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.471426 31915 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.473771 30824 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:08.473012 31915 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:08.527908 31887 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.533178 31729 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.563869 31072 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:08.568379 31950 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20250902 01:58:08.575505 30924 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.584353 30918 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.588735 31860 raft_consensus.cc:1062] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:08.590756 31860 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.605261 30918 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.608863 31729 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.609154 30821 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.606591 31948 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250902 01:58:08.610638 30997 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.613636 31948 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.615775 31901 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.640836 31000 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.642089 31901 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.644694 31729 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.647225 30918 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.652895 31730 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.653538 31953 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250902 01:58:08.658074 31953 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:08.667172 31887 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.668375 30918 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.671470 31730 raft_consensus.cc:2953] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.673797 30924 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.679965 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b03cb35549a346d28baa21d708d665e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:08.688441 31070 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.693631 31729 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:08.694950 30924 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:08.696141 31948 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:08.697948 31948 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:08.702203 31901 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.701833 30994 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.703106 31729 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.716423 30823 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.720657 30822 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:08.732693 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet b03cb35549a346d28baa21d708d665e2 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:08.738554 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ede06f18c6574c57857475151e3767d2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:08.740465 30924 raft_consensus.cc:2953] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.737175 30822 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.758948 31966 tablet_replica.cc:331] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:08.759976 31966 raft_consensus.cc:2241] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:08.760694 31966 raft_consensus.cc:2270] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:08.764798 31070 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.766254 31966 ts_tablet_manager.cc:1905] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:08.771502 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet ede06f18c6574c57857475151e3767d2 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:08.773191 30994 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.774479 31901 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:08.775243 31729 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.776439 31941 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:08.781142 31967 tablet_replica.cc:331] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:08.782236 31967 raft_consensus.cc:2241] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:08.783311 31967 raft_consensus.cc:2270] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:08.783881 31941 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.790478 31729 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.796247 31883 raft_consensus.cc:1062] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:08.798157 31883 consensus_queue.cc:237] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.803925 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0692c5029b5e4ab088324d038ff964be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:08.805524 31729 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.806638 31069 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.808001 31919 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.808998 30997 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:08.788929 30994 raft_consensus.cc:2953] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.811345 31919 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:08.812500 31883 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:08.807720 30820 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:08.819272 31967 ts_tablet_manager.cc:1905] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:08.825457 31966 ts_tablet_manager.cc:1918] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:08.826550 31966 log.cc:1199] T b03cb35549a346d28baa21d708d665e2 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/b03cb35549a346d28baa21d708d665e2
I20250902 01:58:08.835273 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet b03cb35549a346d28baa21d708d665e2 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:08.845252 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 0692c5029b5e4ab088324d038ff964be with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:08.855723 31967 ts_tablet_manager.cc:1918] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20250902 01:58:08.856340 31967 log.cc:1199] T ede06f18c6574c57857475151e3767d2 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/ede06f18c6574c57857475151e3767d2
I20250902 01:58:08.858089 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet ede06f18c6574c57857475151e3767d2 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:08.858318 31971 tablet_replica.cc:331] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:08.859350 31971 raft_consensus.cc:2241] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:08.862005 31971 raft_consensus.cc:2270] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:08.866133 31971 ts_tablet_manager.cc:1905] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250902 01:58:08.870666 31968 log.cc:927] Time spent T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20250902 01:58:08.877038 31919 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.889674 31729 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.892077 30997 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.892997 31971 ts_tablet_manager.cc:1918] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:08.893571 31971 log.cc:1199] T 0692c5029b5e4ab088324d038ff964be P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/0692c5029b5e4ab088324d038ff964be
I20250902 01:58:08.899466 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 0692c5029b5e4ab088324d038ff964be (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:08.899260 30820 catalog_manager.cc:5582] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250902 01:58:08.939630 31973 consensus_queue.cc:1162] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "01f6731ce5044a6a948a5ae04da80e6f" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20250902 01:58:08.939764 31069 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.954272 30923 consensus_queue.cc:237] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:08.967221 30997 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.967653 31729 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:08.968783 31974 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:08.970687 31947 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:08.974248 31974 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.975571 30997 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.978598 31729 raft_consensus.cc:2953] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.982997 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6d30dc90c304472ab905d3af8400b8d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:08.989054 30822 catalog_manager.cc:5582] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:08.997380 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d30dc90c304472ab905d3af8400b8d0 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:08.998109 31966 tablet_replica.cc:331] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:08.998818 31966 raft_consensus.cc:2241] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:08.999471 31966 raft_consensus.cc:2270] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:09.002082 31966 ts_tablet_manager.cc:1905] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:09.011490 31966 ts_tablet_manager.cc:1918] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:09.011917 31966 log.cc:1199] T 6d30dc90c304472ab905d3af8400b8d0 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/6d30dc90c304472ab905d3af8400b8d0
I20250902 01:58:09.013315 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 6d30dc90c304472ab905d3af8400b8d0 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:10.042585 30924 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.046974 31069 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:10.047174 30889 consensus_peers.cc:489] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38649bfc334c4db6921eb3b12130e10d. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:10.047693 30997 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.048018 31883 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.048796 31947 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.056273 31883 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.057567 30997 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.058405 31069 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.072291 30823 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.081773 31069 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:10.090620 31038 consensus_peers.cc:489] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 568a81baa7ef422eb8125c6e8bb9813b. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:10.091928 30997 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.092061 30924 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.093257 31941 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.094764 31941 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.102243 31860 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.104426 30997 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.109304 30923 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.117233 30822 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:10.120568 31069 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.135006 30994 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:10.135121 31038 consensus_peers.cc:489] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f650d4e1b5f47febbda93daccba0e4f. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:10.135295 30923 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.135995 31860 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.136593 31941 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.142928 31860 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.144610 30994 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.147126 30918 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.155432 30924 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.169144 30823 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
W20250902 01:58:10.174860 30889 consensus_peers.cc:489] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0b2dd8f5f814a7d8b48a74020055750. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:10.183373 31069 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.184996 31883 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.174710 30994 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:10.191959 31980 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:10.198518 31947 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.200122 31069 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.208810 30997 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:10.214967 30820 catalog_manager.cc:5582] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:10.424250 31995 ts_tablet_manager.cc:927] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:10.429574 31995 tablet_copy_client.cc:323] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:10.431797 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 38649bfc334c4db6921eb3b12130e10d from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:10.432312 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 38649bfc334c4db6921eb3b12130e10d from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-38649bfc334c4db6921eb3b12130e10d
I20250902 01:58:10.438333 30936 tablet_copy_source_session.cc:215] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:10.441632 31995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38649bfc334c4db6921eb3b12130e10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:10.455065 31995 tablet_copy_client.cc:806] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:10.455682 31995 tablet_copy_client.cc:670] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:10.458904 31995 tablet_copy_client.cc:538] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:10.467379 31995 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:10.487982 31995 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:10.488626 31995 tablet_bootstrap.cc:492] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:10.489117 31995 ts_tablet_manager.cc:1397] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.003s
I20250902 01:58:10.491161 31995 raft_consensus.cc:357] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.491909 31995 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:10.492508 31995 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.496198 31995 ts_tablet_manager.cc:1428] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20250902 01:58:10.497618 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-38649bfc334c4db6921eb3b12130e10d received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:10.498055 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-38649bfc334c4db6921eb3b12130e10d on tablet 38649bfc334c4db6921eb3b12130e10d with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:10.519713 31995 ts_tablet_manager.cc:927] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:10.520957 31995 tablet_copy_client.cc:323] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:10.522482 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 7f650d4e1b5f47febbda93daccba0e4f from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:10.522992 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 7f650d4e1b5f47febbda93daccba0e4f from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-7f650d4e1b5f47febbda93daccba0e4f
I20250902 01:58:10.528473 31084 tablet_copy_source_session.cc:215] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:10.532290 31995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f650d4e1b5f47febbda93daccba0e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:10.543581 31995 tablet_copy_client.cc:806] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:10.544064 31995 tablet_copy_client.cc:670] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:10.547430 31995 tablet_copy_client.cc:538] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:10.553198 31995 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:10.570320 31995 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:10.570981 31995 tablet_bootstrap.cc:492] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:10.571429 31995 ts_tablet_manager.cc:1397] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20250902 01:58:10.573453 31995 raft_consensus.cc:357] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.574151 31995 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:10.574679 31995 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.578534 31995 ts_tablet_manager.cc:1428] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250902 01:58:10.581089 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-7f650d4e1b5f47febbda93daccba0e4f received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:10.581492 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-7f650d4e1b5f47febbda93daccba0e4f on tablet 7f650d4e1b5f47febbda93daccba0e4f with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:10.604861 31995 ts_tablet_manager.cc:927] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:10.606163 31995 tablet_copy_client.cc:323] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:10.607319 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 568a81baa7ef422eb8125c6e8bb9813b from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:10.607705 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 568a81baa7ef422eb8125c6e8bb9813b from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-568a81baa7ef422eb8125c6e8bb9813b
I20250902 01:58:10.613911 31084 tablet_copy_source_session.cc:215] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:10.616022 32002 ts_tablet_manager.cc:927] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:10.616375 31995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 568a81baa7ef422eb8125c6e8bb9813b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:10.621285 32002 tablet_copy_client.cc:323] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:10.622699 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet d0b2dd8f5f814a7d8b48a74020055750 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:10.623152 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet d0b2dd8f5f814a7d8b48a74020055750 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-d0b2dd8f5f814a7d8b48a74020055750
I20250902 01:58:10.628131 30936 tablet_copy_source_session.cc:215] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:10.630653 32002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2dd8f5f814a7d8b48a74020055750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:10.636511 31995 tablet_copy_client.cc:806] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:10.636938 31995 tablet_copy_client.cc:670] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:10.639664 31995 tablet_copy_client.cc:538] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:10.642343 32002 tablet_copy_client.cc:806] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:10.642828 32002 tablet_copy_client.cc:670] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:10.645516 32002 tablet_copy_client.cc:538] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:10.648506 31995 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:10.655961 32002 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:10.670316 31995 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:10.671301 31995 tablet_bootstrap.cc:492] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:10.672109 31995 ts_tablet_manager.cc:1397] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.005s
I20250902 01:58:10.674993 31995 raft_consensus.cc:357] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.675927 31995 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:10.676646 31995 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.679663 31995 ts_tablet_manager.cc:1428] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250902 01:58:10.681643 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-568a81baa7ef422eb8125c6e8bb9813b received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:10.682039 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-568a81baa7ef422eb8125c6e8bb9813b on tablet 568a81baa7ef422eb8125c6e8bb9813b with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:10.684743 32002 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:10.685351 32002 tablet_bootstrap.cc:492] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:10.685817 32002 ts_tablet_manager.cc:1397] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20250902 01:58:10.689107 32002 raft_consensus.cc:357] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.689821 32002 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:10.691051 32002 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:10.693713 32002 ts_tablet_manager.cc:1428] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250902 01:58:10.695613 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d0b2dd8f5f814a7d8b48a74020055750 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:10.696008 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-d0b2dd8f5f814a7d8b48a74020055750 on tablet d0b2dd8f5f814a7d8b48a74020055750 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:10.839108 31730 raft_consensus.cc:1215] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:11.044095 31730 raft_consensus.cc:1215] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:11.136332 31730 raft_consensus.cc:1215] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:11.181092 31730 raft_consensus.cc:1215] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:11.356945 31947 raft_consensus.cc:1062] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:11.358440 31947 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.365746 31730 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.366101 31070 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.366677 31980 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.367177 30994 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.367463 31947 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.368482 31883 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.391111 31883 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.392804 31075 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.397614 30994 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.401103 31730 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.411329 30822 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:11.428493 30918 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.432554 31730 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.436308 30994 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.438262 31980 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.439508 31980 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.445753 31883 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.446820 30994 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.449944 31730 raft_consensus.cc:2953] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.462338 30823 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.470738 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 38649bfc334c4db6921eb3b12130e10d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:11.484133 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 38649bfc334c4db6921eb3b12130e10d with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:11.501999 32018 tablet_replica.cc:331] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:11.509748 32018 raft_consensus.cc:2241] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:11.510341 32018 raft_consensus.cc:2270] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:11.513077 32018 ts_tablet_manager.cc:1905] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:11.525568 32018 ts_tablet_manager.cc:1918] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:11.525854 32018 log.cc:1199] T 38649bfc334c4db6921eb3b12130e10d P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/38649bfc334c4db6921eb3b12130e10d
I20250902 01:58:11.527726 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 38649bfc334c4db6921eb3b12130e10d (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:11.667399 31980 raft_consensus.cc:1062] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:11.669597 31980 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.674849 31729 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.675884 30997 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.675951 31075 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.676923 31980 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.677688 32015 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.678347 32017 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.693707 32016 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.696539 31070 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.703271 31860 raft_consensus.cc:1062] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:11.703862 31728 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.705333 31860 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.719306 30819 catalog_manager.cc:5582] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.730332 30997 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.733392 30924 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.733775 30994 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.738660 32025 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.740118 32024 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.750592 31728 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.748751 30924 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.760686 32025 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.763633 32024 raft_consensus.cc:1062] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:11.768651 30997 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.767766 32024 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.771015 32016 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.779829 32015 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.781984 31730 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.784082 32017 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.782008 30994 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.794142 31728 raft_consensus.cc:2953] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.805073 31727 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.807621 32024 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.811770 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d0b2dd8f5f814a7d8b48a74020055750 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:11.812209 30923 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.814294 32024 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.832032 31860 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.836354 30823 catalog_manager.cc:5582] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.850708 32024 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.857713 31726 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.859817 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2dd8f5f814a7d8b48a74020055750 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:11.860741 32018 tablet_replica.cc:331] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:11.861737 32018 raft_consensus.cc:2241] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:11.862380 32018 raft_consensus.cc:2270] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250902 01:58:11.865818 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 127604us
I20250902 01:58:11.866246 32018 ts_tablet_manager.cc:1905] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:11.878264 30917 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.874017 31728 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.880075 30918 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.890632 31001 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:11.902385 31901 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:11.903723 30823 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.917599 30819 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:11.912807 32018 ts_tablet_manager.cc:1918] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:11.929931 31000 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.932978 31001 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.926663 32018 log.cc:1199] T d0b2dd8f5f814a7d8b48a74020055750 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/d0b2dd8f5f814a7d8b48a74020055750
I20250902 01:58:11.945950 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet d0b2dd8f5f814a7d8b48a74020055750 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:11.963295 31075 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.965637 31070 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:11.976999 30918 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.977288 31726 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.981106 31000 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.980041 31728 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:11.982439 32032 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.982692 31860 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.983481 32024 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.983876 31860 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:11.992902 32025 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.993567 32034 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:11.993070 31000 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:12.000499 31726 raft_consensus.cc:2953] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:12.003047 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 568a81baa7ef422eb8125c6e8bb9813b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:12.005489 31728 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:12.005963 30917 raft_consensus.cc:2953] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:12.012470 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7f650d4e1b5f47febbda93daccba0e4f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:12.018915 30823 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:12.030778 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 568a81baa7ef422eb8125c6e8bb9813b with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:12.033998 30819 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:12.104962 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 67447.1us
I20250902 01:58:12.105332 32038 tablet_replica.cc:331] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:12.113396 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 7f650d4e1b5f47febbda93daccba0e4f with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:12.129271 32039 tablet_replica.cc:331] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:12.130543 32039 raft_consensus.cc:2241] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:12.131026 32038 raft_consensus.cc:2241] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:12.131129 32039 raft_consensus.cc:2270] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:12.131727 32038 raft_consensus.cc:2270] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:12.134646 32038 ts_tablet_manager.cc:1905] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:12.136418 32039 ts_tablet_manager.cc:1905] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:12.194161 32038 ts_tablet_manager.cc:1918] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:12.194701 32038 log.cc:1199] T 568a81baa7ef422eb8125c6e8bb9813b P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/568a81baa7ef422eb8125c6e8bb9813b
I20250902 01:58:12.196278 32039 ts_tablet_manager.cc:1918] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:12.196650 32039 log.cc:1199] T 7f650d4e1b5f47febbda93daccba0e4f P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/7f650d4e1b5f47febbda93daccba0e4f
I20250902 01:58:12.197009 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 568a81baa7ef422eb8125c6e8bb9813b (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:12.199311 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 7f650d4e1b5f47febbda93daccba0e4f (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:13.072618 31070 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.077730 30918 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:13.078153 31001 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250902 01:58:13.078869 31038 consensus_peers.cc:489] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122e9f7b6c1d4295b7d12445ed0116ed. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:13.079100 32025 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:13.079829 32042 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.086536 32042 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.088207 31001 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.088801 30918 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.102030 31001 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.106894 30823 catalog_manager.cc:5582] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.111550 31070 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:13.111874 30918 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:13.112469 30964 consensus_peers.cc:489] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeb7c457a3f741b5b4d057f06363c614. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:13.113106 32026 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:13.115612 32026 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.127503 32026 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.134145 30918 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.137544 31070 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.155467 30918 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.160934 30823 catalog_manager.cc:5582] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.244702 31069 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:13.249457 30889 consensus_peers.cc:489] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02cd1389058944c1adc90f23d0fadb21. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:13.250214 32054 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.260372 31001 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:13.262777 32040 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.264451 32057 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.273231 31001 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.275095 30918 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.267581 31070 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.283269 31069 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:13.284162 30889 consensus_peers.cc:489] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0f24bac96dc42dba9adc1f6679c0d48. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:13.286192 32040 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.286737 31000 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:13.288959 32058 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:13.290732 30820 catalog_manager.cc:5582] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:13.318251 32058 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.319978 30994 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.320272 31069 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.335508 30822 catalog_manager.cc:5582] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:13.628707 32064 ts_tablet_manager.cc:927] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:13.630508 32064 tablet_copy_client.cc:323] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:13.631753 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet aeb7c457a3f741b5b4d057f06363c614 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:13.632197 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet aeb7c457a3f741b5b4d057f06363c614 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-aeb7c457a3f741b5b4d057f06363c614
I20250902 01:58:13.636442 32066 ts_tablet_manager.cc:927] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:13.638134 32066 tablet_copy_client.cc:323] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:13.638648 31011 tablet_copy_source_session.cc:215] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:13.640491 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 122e9f7b6c1d4295b7d12445ed0116ed from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:13.640957 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 122e9f7b6c1d4295b7d12445ed0116ed from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-122e9f7b6c1d4295b7d12445ed0116ed
I20250902 01:58:13.641799 32064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb7c457a3f741b5b4d057f06363c614. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:13.647893 31084 tablet_copy_source_session.cc:215] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:13.650331 32066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122e9f7b6c1d4295b7d12445ed0116ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:13.655936 32064 tablet_copy_client.cc:806] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:13.656473 32064 tablet_copy_client.cc:670] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:13.664803 32064 tablet_copy_client.cc:538] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:13.664975 32066 tablet_copy_client.cc:806] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:13.665670 32066 tablet_copy_client.cc:670] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:13.668744 32066 tablet_copy_client.cc:538] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:13.677577 32064 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:13.677589 32066 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:13.678828 32068 ts_tablet_manager.cc:927] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:13.682300 32068 tablet_copy_client.cc:323] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:13.683729 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 02cd1389058944c1adc90f23d0fadb21 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:13.684162 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 02cd1389058944c1adc90f23d0fadb21 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-02cd1389058944c1adc90f23d0fadb21
I20250902 01:58:13.690104 30936 tablet_copy_source_session.cc:215] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:13.694485 32068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02cd1389058944c1adc90f23d0fadb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:13.697269 32064 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:13.698057 32064 tablet_bootstrap.cc:492] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:13.698675 32064 ts_tablet_manager.cc:1397] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.000s
I20250902 01:58:13.701413 32064 raft_consensus.cc:357] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.702242 32064 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:13.702893 32064 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.715828 32066 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:13.716661 32066 tablet_bootstrap.cc:492] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:13.717396 32066 ts_tablet_manager.cc:1397] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.011s
I20250902 01:58:13.720072 32066 raft_consensus.cc:357] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.720772 32066 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:13.721426 32066 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.737344 32064 ts_tablet_manager.cc:1428] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.038s user 0.011s sys 0.016s
I20250902 01:58:13.740505 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-aeb7c457a3f741b5b4d057f06363c614 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:13.740908 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-aeb7c457a3f741b5b4d057f06363c614 on tablet aeb7c457a3f741b5b4d057f06363c614 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:13.765628 32066 ts_tablet_manager.cc:1428] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.048s user 0.020s sys 0.022s
I20250902 01:58:13.767066 32068 tablet_copy_client.cc:806] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:13.767661 32068 tablet_copy_client.cc:670] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:13.770361 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-122e9f7b6c1d4295b7d12445ed0116ed received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:13.770769 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-122e9f7b6c1d4295b7d12445ed0116ed on tablet 122e9f7b6c1d4295b7d12445ed0116ed with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:13.778535 32068 tablet_copy_client.cc:538] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:13.799635 32068 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:13.800522 32066 ts_tablet_manager.cc:927] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:13.808851 32066 tablet_copy_client.cc:323] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:13.810185 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet f0f24bac96dc42dba9adc1f6679c0d48 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:13.810633 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet f0f24bac96dc42dba9adc1f6679c0d48 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-f0f24bac96dc42dba9adc1f6679c0d48
I20250902 01:58:13.822592 30936 tablet_copy_source_session.cc:215] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:13.831456 32066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f24bac96dc42dba9adc1f6679c0d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:13.840912 32068 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:13.841837 32068 tablet_bootstrap.cc:492] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:13.842532 32068 ts_tablet_manager.cc:1397] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.008s
I20250902 01:58:13.845336 32068 raft_consensus.cc:357] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.846067 32068 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:13.846623 32068 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.848794 32068 ts_tablet_manager.cc:1428] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250902 01:58:13.849886 32066 tablet_copy_client.cc:806] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:13.850383 32066 tablet_copy_client.cc:670] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:13.850462 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-02cd1389058944c1adc90f23d0fadb21 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:13.850991 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-02cd1389058944c1adc90f23d0fadb21 on tablet 02cd1389058944c1adc90f23d0fadb21 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:13.857266 32066 tablet_copy_client.cc:538] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:13.866266 32066 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:13.882643 32066 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:13.883412 32066 tablet_bootstrap.cc:492] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:13.883921 32066 ts_tablet_manager.cc:1397] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.008s
I20250902 01:58:13.885854 32066 raft_consensus.cc:357] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.886667 32066 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:13.887215 32066 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:13.889223 32066 ts_tablet_manager.cc:1428] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250902 01:58:13.890595 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-f0f24bac96dc42dba9adc1f6679c0d48 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:13.890961 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-f0f24bac96dc42dba9adc1f6679c0d48 on tablet f0f24bac96dc42dba9adc1f6679c0d48 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:14.038353 31726 raft_consensus.cc:1215] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:14.117476 31726 raft_consensus.cc:1215] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250902 01:58:14.196621 31726 raft_consensus.cc:1215] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:14.367326 31726 raft_consensus.cc:1215] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:14.610746 32077 raft_consensus.cc:1062] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:14.613121 32077 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.618960 32026 raft_consensus.cc:1062] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:14.621301 32026 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.631006 30924 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:14.631983 31726 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.633271 31069 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.633908 31726 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:14.634533 30994 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:14.635180 32006 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.636781 32077 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.638906 32006 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.648324 32077 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.650279 32025 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.655148 32054 raft_consensus.cc:1062] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:14.660102 32054 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.667109 32042 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.671424 31726 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.693724 30918 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.699771 30917 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.700314 30822 catalog_manager.cc:5582] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:14.701027 32080 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:14.712154 31001 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.715736 31726 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.717183 31069 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.718432 32054 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:14.719628 32058 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.726292 30994 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.728538 32006 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.729079 32054 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.731155 30917 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.741782 31726 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.757079 32058 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.760056 31001 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.773468 31728 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.760089 31070 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.792093 31069 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.810695 30819 catalog_manager.cc:5582] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.812134 30819 catalog_manager.cc:5582] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.806046 31070 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.819818 30918 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:14.821208 32042 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:14.823688 31726 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:14.829445 32042 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:14.829427 31001 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.834460 32040 raft_consensus.cc:1062] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:14.835244 30920 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.838056 31729 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:14.839243 32006 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:14.836838 32040 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.833683 32025 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.844201 31072 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:14.848992 32026 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.856706 32006 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:14.858282 31729 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.860483 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 122e9f7b6c1d4295b7d12445ed0116ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:14.860647 31728 raft_consensus.cc:2953] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.862308 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet aeb7c457a3f741b5b4d057f06363c614 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:14.867182 30918 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:14.874692 31069 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.877607 31730 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:14.877915 31726 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.880383 31070 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:14.881858 32054 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250902 01:58:14.885011 31001 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:14.886183 32054 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.889467 32058 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:14.889557 32054 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.888510 30822 catalog_manager.cc:5582] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.896126 32058 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.897222 31069 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.913877 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 122e9f7b6c1d4295b7d12445ed0116ed with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:14.919898 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 02cd1389058944c1adc90f23d0fadb21 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250902 01:58:14.920333 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51128.3us
I20250902 01:58:14.926535 30820 catalog_manager.cc:5582] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:14.931098 32102 tablet_replica.cc:331] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:14.932144 32102 raft_consensus.cc:2241] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:14.939618 30822 catalog_manager.cc:5582] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.948769 31069 raft_consensus.cc:2953] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.950565 32102 raft_consensus.cc:2270] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:14.954478 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet aeb7c457a3f741b5b4d057f06363c614 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:14.961468 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 02cd1389058944c1adc90f23d0fadb21 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:14.972539 32102 ts_tablet_manager.cc:1905] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:14.973783 32106 tablet_replica.cc:331] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:14.974820 32106 raft_consensus.cc:2241] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:14.977437 32095 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:14.978504 32106 raft_consensus.cc:2270] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:14.989965 32106 ts_tablet_manager.cc:1905] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:14.992336 32102 ts_tablet_manager.cc:1918] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20250902 01:58:14.992738 32102 log.cc:1199] T 122e9f7b6c1d4295b7d12445ed0116ed P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/122e9f7b6c1d4295b7d12445ed0116ed
I20250902 01:58:14.994218 32102 tablet_replica.cc:331] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:14.999758 31729 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.001375 32102 raft_consensus.cc:2241] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:15.002043 32102 raft_consensus.cc:2270] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:15.003381 31726 raft_consensus.cc:2953] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.005652 32102 ts_tablet_manager.cc:1905] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:14.995751 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 122e9f7b6c1d4295b7d12445ed0116ed (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:15.009390 31001 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:14.997959 32058 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.019676 32102 ts_tablet_manager.cc:1918] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:15.020124 32102 log.cc:1199] T 02cd1389058944c1adc90f23d0fadb21 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/02cd1389058944c1adc90f23d0fadb21
I20250902 01:58:15.024031 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 02cd1389058944c1adc90f23d0fadb21 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:15.030090 30819 catalog_manager.cc:5582] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.030532 32106 ts_tablet_manager.cc:1918] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:15.033146 31069 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.035497 32106 log.cc:1199] T aeb7c457a3f741b5b4d057f06363c614 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/aeb7c457a3f741b5b4d057f06363c614
I20250902 01:58:15.037173 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet aeb7c457a3f741b5b4d057f06363c614 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:15.054840 30920 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:15.059634 31726 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:15.060815 31001 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:15.062326 32095 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:15.063072 32104 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:15.072947 31001 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.071460 32095 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.080013 31726 raft_consensus.cc:2953] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:15.090905 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f0f24bac96dc42dba9adc1f6679c0d48 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:15.092126 30822 catalog_manager.cc:5582] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:15.103178 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet f0f24bac96dc42dba9adc1f6679c0d48 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:15.106441 32108 tablet_replica.cc:331] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:15.107329 32108 raft_consensus.cc:2241] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:15.108047 32108 raft_consensus.cc:2270] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:15.110745 32108 ts_tablet_manager.cc:1905] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:15.121951 32108 ts_tablet_manager.cc:1918] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:15.122283 32108 log.cc:1199] T f0f24bac96dc42dba9adc1f6679c0d48 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/f0f24bac96dc42dba9adc1f6679c0d48
I20250902 01:58:15.123574 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet f0f24bac96dc42dba9adc1f6679c0d48 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:16.177702 31069 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.189054 30994 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:16.190009 31038 consensus_peers.cc:489] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2925593648194df5b7741d448a047401. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:16.190959 30918 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:16.191088 32077 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:16.192812 32109 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:16.201118 32077 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.203871 30994 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.207082 30918 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.230255 31070 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.232574 30819 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
W20250902 01:58:16.253803 31038 consensus_peers.cc:489] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50a1d33046ce49cd891b784fbe726d64. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:16.254228 30918 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:16.255278 31001 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:16.256319 32042 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:16.257133 32109 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:16.270051 32109 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.273262 31001 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.290805 30822 catalog_manager.cc:5582] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.297427 30918 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.306565 31069 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:16.325783 31038 consensus_peers.cc:489] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2d2c9c23b6145c9b6601f7e78f1f43a. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:16.325985 31001 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:16.327142 32124 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:16.328095 30918 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:16.329226 32124 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:16.337385 32077 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.340066 31001 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.340474 30920 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.353713 30994 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.354542 30819 catalog_manager.cc:5582] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:16.369762 30964 consensus_peers.cc:489] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 349593d2d3574cd6ab281c105756f435. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:16.372629 31069 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:16.371713 30920 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:16.374224 32081 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:16.381469 32026 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:16.388835 31070 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.394429 32026 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.410178 30918 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.421257 30822 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:16.571929 32134 ts_tablet_manager.cc:927] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:16.573791 32134 tablet_copy_client.cc:323] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:16.575264 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 2925593648194df5b7741d448a047401 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:16.575824 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 2925593648194df5b7741d448a047401 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-2925593648194df5b7741d448a047401
I20250902 01:58:16.582926 31084 tablet_copy_source_session.cc:215] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:16.585837 32134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925593648194df5b7741d448a047401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:16.601346 32134 tablet_copy_client.cc:806] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:16.601935 32134 tablet_copy_client.cc:670] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:16.605597 32134 tablet_copy_client.cc:538] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:16.611516 32134 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:16.628773 32134 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:16.629601 32134 tablet_bootstrap.cc:492] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:16.630260 32134 ts_tablet_manager.cc:1397] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.008s
I20250902 01:58:16.632630 32134 raft_consensus.cc:357] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.633224 32134 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:16.633713 32134 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.638612 32134 ts_tablet_manager.cc:1428] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20250902 01:58:16.640699 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-2925593648194df5b7741d448a047401 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:16.641134 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-2925593648194df5b7741d448a047401 on tablet 2925593648194df5b7741d448a047401 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:16.651337 32134 ts_tablet_manager.cc:927] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:16.654573 32134 tablet_copy_client.cc:323] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:16.655678 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 50a1d33046ce49cd891b784fbe726d64 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:16.656103 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 50a1d33046ce49cd891b784fbe726d64 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-50a1d33046ce49cd891b784fbe726d64
I20250902 01:58:16.661803 31084 tablet_copy_source_session.cc:215] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:16.664203 32134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50a1d33046ce49cd891b784fbe726d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:16.675379 32134 tablet_copy_client.cc:806] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:16.675869 32134 tablet_copy_client.cc:670] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:16.679129 32134 tablet_copy_client.cc:538] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:16.687852 32134 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:16.705425 32134 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:16.706049 32134 tablet_bootstrap.cc:492] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:16.706562 32134 ts_tablet_manager.cc:1397] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.011s
I20250902 01:58:16.708837 32134 raft_consensus.cc:357] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.709539 32134 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:16.710071 32134 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.712392 32134 ts_tablet_manager.cc:1428] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250902 01:58:16.713980 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-50a1d33046ce49cd891b784fbe726d64 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:16.714360 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-50a1d33046ce49cd891b784fbe726d64 on tablet 50a1d33046ce49cd891b784fbe726d64 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:16.845149 32134 ts_tablet_manager.cc:927] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:16.846622 32134 tablet_copy_client.cc:323] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:16.848157 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet 349593d2d3574cd6ab281c105756f435 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:16.848616 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet 349593d2d3574cd6ab281c105756f435 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-349593d2d3574cd6ab281c105756f435
I20250902 01:58:16.854063 31011 tablet_copy_source_session.cc:215] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:16.856622 32134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349593d2d3574cd6ab281c105756f435. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:16.868980 32134 tablet_copy_client.cc:806] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:16.869446 32134 tablet_copy_client.cc:670] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:16.872364 32134 tablet_copy_client.cc:538] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:16.878804 32134 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:16.884693 32141 ts_tablet_manager.cc:927] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:16.888339 32141 tablet_copy_client.cc:323] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:16.889998 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet a2d2c9c23b6145c9b6601f7e78f1f43a from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:16.890621 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet a2d2c9c23b6145c9b6601f7e78f1f43a from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-a2d2c9c23b6145c9b6601f7e78f1f43a
I20250902 01:58:16.897637 31084 tablet_copy_source_session.cc:215] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:16.897948 32134 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:16.898633 32134 tablet_bootstrap.cc:492] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:16.899114 32134 ts_tablet_manager.cc:1397] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.003s
I20250902 01:58:16.901347 32134 raft_consensus.cc:357] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.902112 32134 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:16.902642 32134 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.903667 32141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d2c9c23b6145c9b6601f7e78f1f43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:16.904513 32134 ts_tablet_manager.cc:1428] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.005s user 0.003s sys 0.005s
I20250902 01:58:16.905678 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-349593d2d3574cd6ab281c105756f435 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:16.905977 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-349593d2d3574cd6ab281c105756f435 on tablet 349593d2d3574cd6ab281c105756f435 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:16.923616 32141 tablet_copy_client.cc:806] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:16.924149 32141 tablet_copy_client.cc:670] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:16.927451 32141 tablet_copy_client.cc:538] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:16.937108 32141 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:16.962107 32141 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:16.963079 32141 tablet_bootstrap.cc:492] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:16.963680 32141 ts_tablet_manager.cc:1397] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20250902 01:58:16.966053 32141 raft_consensus.cc:357] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.966910 32141 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:16.967640 32141 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:16.970350 32141 ts_tablet_manager.cc:1428] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250902 01:58:16.972707 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-a2d2c9c23b6145c9b6601f7e78f1f43a received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:16.973801 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-a2d2c9c23b6145c9b6601f7e78f1f43a on tablet a2d2c9c23b6145c9b6601f7e78f1f43a with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:17.077467 31729 raft_consensus.cc:1215] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:17.124907 31729 raft_consensus.cc:1215] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:17.225795 31726 raft_consensus.cc:1215] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250902 01:58:17.396632 31729 raft_consensus.cc:1215] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:17.597637 32124 raft_consensus.cc:1062] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:17.599668 32124 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.605695 30994 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.605877 30920 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.605887 31729 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.606989 32109 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.608844 32124 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.611057 32042 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.627774 32109 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.632856 30994 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.637573 31729 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.639238 30920 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.645856 30822 catalog_manager.cc:5582] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:17.671518 31069 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.679224 31729 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:17.680491 32109 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.685014 30994 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:17.686151 32042 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.687794 32124 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.688493 31729 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.692306 31000 raft_consensus.cc:2953] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.695565 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 50a1d33046ce49cd891b784fbe726d64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:17.698877 30820 catalog_manager.cc:5582] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.712867 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 50a1d33046ce49cd891b784fbe726d64 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:17.726239 32155 tablet_replica.cc:331] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:17.727368 32155 raft_consensus.cc:2241] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:17.727958 32155 raft_consensus.cc:2270] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:17.731057 32155 ts_tablet_manager.cc:1905] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:17.743352 32155 ts_tablet_manager.cc:1918] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:17.743758 32155 log.cc:1199] T 50a1d33046ce49cd891b784fbe726d64 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/50a1d33046ce49cd891b784fbe726d64
I20250902 01:58:17.745169 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 50a1d33046ce49cd891b784fbe726d64 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:17.750133 32124 raft_consensus.cc:1062] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:17.752231 32124 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.757802 30920 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.757818 31729 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.757899 31000 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.759141 32157 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.761121 32124 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250902 01:58:17.764345 32042 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.777534 32109 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.785131 31729 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.785090 30994 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.790971 30918 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.802758 30822 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:17.814543 32109 raft_consensus.cc:1062] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:17.816540 32109 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.828720 30994 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.832742 30918 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.836123 31729 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:17.836580 31069 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.838126 32157 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:17.840315 32130 raft_consensus.cc:1062] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:17.853170 31729 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:17.852931 32130 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.854431 32157 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.854103 32124 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250902 01:58:17.857851 32042 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:17.859740 32042 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.862175 31730 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.864126 30920 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:17.865155 32130 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.867128 31729 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:17.870450 32130 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.881368 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2925593648194df5b7741d448a047401 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:17.881007 30822 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:17.889514 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 53016.7us
I20250902 01:58:17.892627 31075 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:17.894804 30994 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:17.898854 32130 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.900331 32042 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:17.903925 32042 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.905014 31726 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.908423 32145 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.910183 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 2925593648194df5b7741d448a047401 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:17.915805 32155 tablet_replica.cc:331] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:17.916117 31000 raft_consensus.cc:2953] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.916991 32155 raft_consensus.cc:2241] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:17.917934 32155 raft_consensus.cc:2270] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:17.910812 30920 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.929332 32155 ts_tablet_manager.cc:1905] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:17.942348 31000 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.942653 30823 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:17.943868 31726 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.952927 30920 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.969960 30822 catalog_manager.cc:5582] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.976845 31001 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:17.980707 31069 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.987349 31729 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:17.988227 30918 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:17.986549 32155 ts_tablet_manager.cc:1918] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:17.989058 32155 log.cc:1199] T 2925593648194df5b7741d448a047401 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/2925593648194df5b7741d448a047401
I20250902 01:58:17.990391 32162 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:58:17.992666 32162 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:17.998137 32145 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:17.995960 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 2925593648194df5b7741d448a047401 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:18.000581 30918 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.003289 31729 raft_consensus.cc:2953] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.006940 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 349593d2d3574cd6ab281c105756f435 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:18.015698 31069 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:18.020584 30820 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 4 to 5, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.021283 31726 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:18.025518 30918 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:18.027339 32169 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250902 01:58:18.029479 32157 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:18.030905 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 349593d2d3574cd6ab281c105756f435 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:18.036266 32109 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.044384 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a2d2c9c23b6145c9b6601f7e78f1f43a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:18.046626 30918 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.049654 32172 tablet_replica.cc:331] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:18.058151 32172 raft_consensus.cc:2241] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:18.058743 32172 raft_consensus.cc:2270] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:18.062068 30819 catalog_manager.cc:5582] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:18.064848 32172 ts_tablet_manager.cc:1905] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:18.065038 31729 raft_consensus.cc:2953] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:18.075467 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet a2d2c9c23b6145c9b6601f7e78f1f43a with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:18.078388 32172 ts_tablet_manager.cc:1918] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20250902 01:58:18.078831 32172 log.cc:1199] T 349593d2d3574cd6ab281c105756f435 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/349593d2d3574cd6ab281c105756f435
I20250902 01:58:18.081189 32173 tablet_replica.cc:331] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:18.081672 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 349593d2d3574cd6ab281c105756f435 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:18.082412 32173 raft_consensus.cc:2241] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:18.082885 32173 raft_consensus.cc:2270] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:18.085472 32173 ts_tablet_manager.cc:1905] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:18.096354 32173 ts_tablet_manager.cc:1918] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:18.096633 32173 log.cc:1199] T a2d2c9c23b6145c9b6601f7e78f1f43a P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/a2d2c9c23b6145c9b6601f7e78f1f43a
I20250902 01:58:18.097882 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet a2d2c9c23b6145c9b6601f7e78f1f43a (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:19.113874 31075 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:19.119666 30918 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:19.119853 31038 consensus_peers.cc:489] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eecf760e0123488fb797b4457e0888d7. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:19.120543 31001 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:19.120819 32157 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.122244 32109 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.128132 32157 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.129413 30920 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.130748 31000 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.144594 30819 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.146811 30920 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:19.163372 30889 consensus_peers.cc:489] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f27fd54f16465fa39cc86d8d6c07bc. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:19.164191 31000 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:19.164544 31075 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:19.165768 32181 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.166996 32182 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.180313 32148 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.189047 31000 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.190922 31075 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.207558 30822 catalog_manager.cc:5582] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.208374 30920 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:19.214623 31075 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:19.214802 31000 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:19.215616 30889 consensus_peers.cc:489] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818cd33c83fe427a981bf0c807af5b14. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:19.216744 32182 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.218822 32181 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.234684 32148 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.236176 31069 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.237717 31001 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.254648 30819 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.257982 31001 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:19.264999 31069 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:19.265173 30964 consensus_peers.cc:489] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c93c2499f7743579a074e5f2646ad2d. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:19.265909 30920 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:19.266224 32145 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.267045 32162 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:19.283113 32162 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.287901 31069 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.289753 30918 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:19.303565 30819 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:19.679196 32195 ts_tablet_manager.cc:927] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:19.688874 32196 ts_tablet_manager.cc:927] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:19.695492 32195 tablet_copy_client.cc:323] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:19.696552 32196 tablet_copy_client.cc:323] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:19.697188 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 69f27fd54f16465fa39cc86d8d6c07bc from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:19.697741 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 69f27fd54f16465fa39cc86d8d6c07bc from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-69f27fd54f16465fa39cc86d8d6c07bc
I20250902 01:58:19.698589 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet eecf760e0123488fb797b4457e0888d7 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:19.699084 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet eecf760e0123488fb797b4457e0888d7 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-eecf760e0123488fb797b4457e0888d7
I20250902 01:58:19.704777 31084 tablet_copy_source_session.cc:215] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:19.704939 30936 tablet_copy_source_session.cc:215] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:19.713644 32195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f27fd54f16465fa39cc86d8d6c07bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:19.754510 32195 tablet_copy_client.cc:806] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:19.755097 32195 tablet_copy_client.cc:670] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:19.766656 32196 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eecf760e0123488fb797b4457e0888d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:19.784842 32196 tablet_copy_client.cc:806] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:19.785445 32196 tablet_copy_client.cc:670] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:19.911697 32200 ts_tablet_manager.cc:927] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:19.949596 32211 ts_tablet_manager.cc:927] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:19.956514 32200 tablet_copy_client.cc:323] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:19.958287 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet 0c93c2499f7743579a074e5f2646ad2d from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:19.958801 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet 0c93c2499f7743579a074e5f2646ad2d from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-0c93c2499f7743579a074e5f2646ad2d
W20250902 01:58:19.959878 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 58384.9us
I20250902 01:58:19.958312 32195 tablet_copy_client.cc:538] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:19.970446 31011 tablet_copy_source_session.cc:215] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:19.980960 32196 tablet_copy_client.cc:538] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:19.981396 32211 tablet_copy_client.cc:323] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:19.982775 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 818cd33c83fe427a981bf0c807af5b14 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:19.981053 32200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c93c2499f7743579a074e5f2646ad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:19.983206 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 818cd33c83fe427a981bf0c807af5b14 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-818cd33c83fe427a981bf0c807af5b14
W20250902 01:58:19.984856 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 44746.5us
I20250902 01:58:19.990972 30936 tablet_copy_source_session.cc:215] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:19.992995 32195 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:19.999656 32211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818cd33c83fe427a981bf0c807af5b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:20.005237 32200 tablet_copy_client.cc:806] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:20.005698 32200 tablet_copy_client.cc:670] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:20.011298 32200 tablet_copy_client.cc:538] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:20.024505 32196 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:20.026814 32200 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:20.045325 32211 tablet_copy_client.cc:806] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:20.045857 32211 tablet_copy_client.cc:670] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:20.050570 32211 tablet_copy_client.cc:538] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:20.063874 32196 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:20.064602 32211 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:20.064762 32196 tablet_bootstrap.cc:492] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:20.065387 32196 ts_tablet_manager.cc:1397] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.008s
I20250902 01:58:20.066087 32200 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:20.060158 32195 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:20.067600 32195 tablet_bootstrap.cc:492] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:20.068347 32195 ts_tablet_manager.cc:1397] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.076s user 0.029s sys 0.010s
I20250902 01:58:20.071763 32200 tablet_bootstrap.cc:492] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:20.072373 32200 ts_tablet_manager.cc:1397] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.012s
I20250902 01:58:20.074815 32200 raft_consensus.cc:357] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.075464 32200 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:20.076043 32200 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.070730 32195 raft_consensus.cc:357] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.078125 32195 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:20.078665 32195 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.082109 32196 raft_consensus.cc:357] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.083565 32196 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:20.094069 32196 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.098089 32200 ts_tablet_manager.cc:1428] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.025s user 0.007s sys 0.005s
I20250902 01:58:20.100723 32195 ts_tablet_manager.cc:1428] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.032s user 0.011s sys 0.003s
I20250902 01:58:20.103232 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-69f27fd54f16465fa39cc86d8d6c07bc received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:20.103628 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-69f27fd54f16465fa39cc86d8d6c07bc on tablet 69f27fd54f16465fa39cc86d8d6c07bc with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.105530 32196 ts_tablet_manager.cc:1428] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.040s user 0.000s sys 0.006s
I20250902 01:58:20.105595 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-0c93c2499f7743579a074e5f2646ad2d received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:20.106021 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-0c93c2499f7743579a074e5f2646ad2d on tablet 0c93c2499f7743579a074e5f2646ad2d with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.107263 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-eecf760e0123488fb797b4457e0888d7 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:20.107661 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-eecf760e0123488fb797b4457e0888d7 on tablet eecf760e0123488fb797b4457e0888d7 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.117075 31726 raft_consensus.cc:1215] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:20.119199 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:20.119987 32211 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:20.120834 32211 tablet_bootstrap.cc:492] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:20.121449 32211 ts_tablet_manager.cc:1397] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.016s
I20250902 01:58:20.125702 32211 raft_consensus.cc:357] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.126442 32211 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:20.126883 32211 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:20.129117 32211 ts_tablet_manager.cc:1428] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250902 01:58:20.132567 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-818cd33c83fe427a981bf0c807af5b14 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:20.132900 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-818cd33c83fe427a981bf0c807af5b14 on tablet 818cd33c83fe427a981bf0c807af5b14 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.187412 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:20.187942 31069 raft_consensus.cc:604] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:20.190043 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:20.190510 31069 raft_consensus.cc:604] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.205292 30924 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:20.205703 30924 raft_consensus.cc:604] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:20.227385 31000 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:20.227788 31000 raft_consensus.cc:604] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.231422 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:20.231885 31069 raft_consensus.cc:604] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:20.235365 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:20.235812 31069 raft_consensus.cc:604] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:20.237743 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ede06f18c6574c57857475151e3767d2"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:20.238193 31069 raft_consensus.cc:604] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:20.239101 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20250902 01:58:20.240563 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:20.241413 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:20.272231 31730 raft_consensus.cc:1215] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:20.441942 32148 raft_consensus.cc:991] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:20.442409 32148 raft_consensus.cc:1079] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:20.443595 31070 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:20.443967 31070 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.444229 31070 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:20.446064 31730 raft_consensus.cc:1215] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:20.447642 32109 raft_consensus.cc:991] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.448027 32109 raft_consensus.cc:1079] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.452260 31730 raft_consensus.cc:1215] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:20.453302 31001 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:20.453740 31001 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.454030 31001 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.458338 31070 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.460475 31070 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:20.461102 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.461689 30924 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:20.461942 30924 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:20.462550 30924 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.463521 30924 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:20.472136 30924 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:20.473158 31039 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:20.474202 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:20.473718 31001 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.474720 31000 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:20.475278 32204 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:20.475759 32204 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.476450 32204 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.476912 31001 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:20.477923 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:20.478410 31070 raft_consensus.cc:3053] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:20.478653 31070 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.479503 31070 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.480497 31000 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:20.483916 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.488267 30924 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.489511 31070 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.491590 30822 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.497531 31070 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.501044 32162 raft_consensus.cc:991] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.501485 32162 raft_consensus.cc:1079] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.503090 31730 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:56978
I20250902 01:58:20.503566 31730 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.503866 31730 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.513633 30967 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:20.514659 32213 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:20.515108 32213 raft_consensus.cc:695] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:20.515882 32213 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.517481 32201 raft_consensus.cc:991] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.518601 32201 raft_consensus.cc:1079] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.511793 31730 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.520213 31726 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:20.520670 31726 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.520967 31726 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.521454 31730 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:20.527061 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:20.527581 31000 raft_consensus.cc:3053] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:20.527854 31000 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:20.528406 31000 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.536084 31000 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.541839 31726 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.543854 31726 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:20.545022 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:20.545485 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:20.545743 30998 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.545985 31070 raft_consensus.cc:3053] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:20.546329 31070 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.546875 31070 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.548235 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.548688 30920 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.553045 31070 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.558138 31000 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:20.559029 31692 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:20.559968 32220 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:20.560459 32220 raft_consensus.cc:695] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:20.561321 32109 raft_consensus.cc:1062] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:20.561283 32220 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.564936 30920 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:20.568203 31070 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:20.569339 31695 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:20.570542 32219 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:20.571141 32219 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:20.571972 32219 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.582836 32201 raft_consensus.cc:991] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.565925 32109 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.583425 32201 raft_consensus.cc:1079] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.585489 31000 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:20.586787 31000 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.587100 31000 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.591746 31000 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.498632 30924 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.596262 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.597759 30924 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.596899 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:20.599731 30998 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:20.600178 31069 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:20.600521 31069 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.601477 31069 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.602329 31069 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.597266 31000 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:20.607467 31069 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.608933 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:20.609853 32192 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:20.610330 32192 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:20.611125 32192 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.614794 31730 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.617966 30920 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.619761 32201 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250902 01:58:20.623272 32201 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:20.626781 30998 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.629662 30924 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.634424 32199 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.638548 32199 raft_consensus.cc:991] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.640925 32199 raft_consensus.cc:1079] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:20.644035 32204 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.651695 30819 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.652190 30820 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:20.660512 31000 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:20.661280 31000 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.660858 31730 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.661609 31000 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.668545 30924 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.664397 30998 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.675195 31000 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.677273 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.677791 30920 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.677932 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:20.678458 31069 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:20.678720 31069 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.679374 31069 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.682255 31069 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:20.685804 30920 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.686976 30965 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:20.687501 31000 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:20.688113 32192 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:20.687039 31069 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:20.688568 32192 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:20.690002 32192 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:20.696563 30823 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.706796 30822 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.708891 32204 raft_consensus.cc:991] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.709308 32199 raft_consensus.cc:1079] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:20.711155 31726 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ede06f18c6574c57857475151e3767d2"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:20.711711 31726 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:20.712067 31726 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:20.720942 31726 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.720865 30820 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:20.722280 31070 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.722532 30820 catalog_manager.cc:5582] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:20.723874 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:20.726166 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:20.726997 30920 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:20.727082 31069 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:20.727380 31069 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:20.722671 31726 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:20.730244 30998 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:20.730355 31728 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:20.731770 32204 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:20.732625 30920 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:20.733600 31693 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:20.733422 31069 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.733675 32201 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:20.734402 32220 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:20.734591 31069 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:20.734870 32220 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:20.735666 32220 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.745112 32109 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.747524 30997 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.756006 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet eecf760e0123488fb797b4457e0888d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:20.757347 31069 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:20.756773 31728 raft_consensus.cc:2953] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.761476 30820 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.772887 30819 catalog_manager.cc:5582] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 4 to 5, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 5 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.789422 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet eecf760e0123488fb797b4457e0888d7 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:20.798784 32241 tablet_replica.cc:331] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:20.800256 32241 raft_consensus.cc:2241] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:20.800937 32241 raft_consensus.cc:2270] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:20.805142 32241 ts_tablet_manager.cc:1905] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:20.817274 32241 ts_tablet_manager.cc:1918] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:20.817574 32241 log.cc:1199] T eecf760e0123488fb797b4457e0888d7 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/eecf760e0123488fb797b4457e0888d7
I20250902 01:58:20.819128 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet eecf760e0123488fb797b4457e0888d7 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:20.833804 32192 raft_consensus.cc:1062] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:20.835866 32192 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.841542 31728 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.842741 32162 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.843194 31069 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.844550 30918 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.844535 32213 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.846166 32213 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.858573 32192 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.870720 30918 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.863986 31069 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.876295 30822 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:20.879652 32207 raft_consensus.cc:1062] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
W20250902 01:58:20.880424 31096 debug-util.cc:398] Leaking SignalData structure 0x7b08001abf60 after lost signal to thread 29786
W20250902 01:58:20.881353 31096 debug-util.cc:398] Leaking SignalData structure 0x7b080017b080 after lost signal to thread 30858
I20250902 01:58:20.882488 32207 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:20.885890 31096 debug-util.cc:398] Leaking SignalData structure 0x7b080025e100 after lost signal to thread 31754
I20250902 01:58:20.887001 30920 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:20.888229 32201 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.898743 30998 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:20.906505 31726 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.875030 31728 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.908628 32058 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.908943 31070 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.910238 32058 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.919488 31070 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:20.920814 32192 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:20.927790 31726 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:20.933876 32162 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:20.935173 32192 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.937492 32058 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.944938 31726 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.945616 31075 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.950316 30924 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:20.951134 31070 raft_consensus.cc:2953] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.949306 30998 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:20.960956 32213 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:20.965744 32148 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.972384 31072 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:20.973687 32220 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:20.976092 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0c93c2499f7743579a074e5f2646ad2d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:20.979005 30822 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.988335 31728 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:20.992830 30997 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:20.994107 32157 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:20.994282 30819 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.014058 32207 raft_consensus.cc:1062] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
W20250902 01:58:21.014370 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 52306.7us
I20250902 01:58:21.012631 30998 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.016275 30924 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.016469 32207 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.024367 31075 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:21.026175 32148 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:21.028765 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c93c2499f7743579a074e5f2646ad2d with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:21.029755 32241 tablet_replica.cc:331] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:21.030664 32241 raft_consensus.cc:2241] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:21.031311 32241 raft_consensus.cc:2270] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:21.032809 32148 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.036784 32241 ts_tablet_manager.cc:1905] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:21.040767 31726 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:21.045439 30924 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:21.046720 32217 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:21.045873 31070 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.063910 31069 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:21.065155 32203 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:21.081491 30998 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:21.082752 32220 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:21.089636 31726 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:21.103787 32203 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250902 01:58:21.106113 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 818cd33c83fe427a981bf0c807af5b14 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:21.109505 32241 ts_tablet_manager.cc:1918] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:21.110018 32241 log.cc:1199] T 0c93c2499f7743579a074e5f2646ad2d P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/0c93c2499f7743579a074e5f2646ad2d
I20250902 01:58:21.111407 31075 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:21.112483 32058 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:21.116277 31726 raft_consensus.cc:2953] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.115685 32252 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:21.120298 30994 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:21.121603 32219 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:21.125761 30998 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:21.134184 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 0c93c2499f7743579a074e5f2646ad2d (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:21.136318 30823 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:21.138041 31075 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:21.140342 32252 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:21.153643 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 818cd33c83fe427a981bf0c807af5b14 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:21.183115 32268 tablet_replica.cc:331] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:21.185053 32268 raft_consensus.cc:2241] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:21.187592 32268 raft_consensus.cc:2270] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:21.191886 32268 ts_tablet_manager.cc:1905] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:21.200017 31069 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:21.206900 32268 ts_tablet_manager.cc:1918] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:21.207327 32268 log.cc:1199] T 818cd33c83fe427a981bf0c807af5b14 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/818cd33c83fe427a981bf0c807af5b14
I20250902 01:58:21.208016 32252 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:21.210279 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 818cd33c83fe427a981bf0c807af5b14 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:21.214677 32058 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:21.228210 30926 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:21.240837 32162 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:21.240497 32058 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.245426 31069 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:21.246392 30926 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:21.249434 32220 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:21.263720 32213 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:21.263978 31728 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.274380 30917 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:21.275863 32219 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:21.277940 30998 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.286571 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:21.297842 31073 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.307752 30819 catalog_manager.cc:5582] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.353193 30917 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.359524 31726 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:21.362777 32203 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:21.366457 30998 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:21.368569 32203 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:21.375867 32203 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.378068 31728 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.379711 30998 raft_consensus.cc:2953] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.392027 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 69f27fd54f16465fa39cc86d8d6c07bc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:21.393187 30823 catalog_manager.cc:5582] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:21.410964 30998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "416020230983423c9f843070a7893658"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:21.412449 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f27fd54f16465fa39cc86d8d6c07bc with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:21.411448 30998 raft_consensus.cc:604] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:21.419384 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:21.419896 30920 raft_consensus.cc:604] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:21.422498 31070 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:21.422976 31070 raft_consensus.cc:604] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:21.425392 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:21.425828 30920 raft_consensus.cc:604] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:21.430018 32276 tablet_replica.cc:331] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:21.431747 31073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:21.437858 31073 raft_consensus.cc:604] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:21.431305 32276 raft_consensus.cc:2241] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:21.441599 31070 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f160859afd5945949c030026bfe3ac61"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:21.442276 31070 raft_consensus.cc:604] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:21.444332 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250902 01:58:21.445720 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:21.446475 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:21.451593 32276 raft_consensus.cc:2270] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:21.458850 32276 ts_tablet_manager.cc:1905] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:21.471804 32276 ts_tablet_manager.cc:1918] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:21.472189 32276 log.cc:1199] T 69f27fd54f16465fa39cc86d8d6c07bc P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/69f27fd54f16465fa39cc86d8d6c07bc
I20250902 01:58:21.473512 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 69f27fd54f16465fa39cc86d8d6c07bc (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:21.513885 32109 raft_consensus.cc:991] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.514281 32199 raft_consensus.cc:1079] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.515502 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f160859afd5945949c030026bfe3ac61"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:21.515916 31728 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.516136 31728 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.520072 31728 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.521536 31728 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:21.522097 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:21.522625 30998 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.522912 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:21.523334 31070 raft_consensus.cc:3053] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.523530 31070 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.524223 31070 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.525149 31070 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.528774 31070 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.529915 31695 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:21.530628 32219 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.531085 32219 raft_consensus.cc:695] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:21.531854 32219 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.534850 32252 raft_consensus.cc:991] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.535265 32252 raft_consensus.cc:1079] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.537737 32203 raft_consensus.cc:991] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.538139 32203 raft_consensus.cc:1079] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.540843 31073 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "416020230983423c9f843070a7893658"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46548
I20250902 01:58:21.541504 31073 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.541800 31069 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:21.542342 31069 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.542654 31069 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.546856 30998 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.549435 31069 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.541803 31073 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.550307 30823 catalog_manager.cc:5582] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.551630 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:21.552098 30920 raft_consensus.cc:3053] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.552330 30920 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:21.552887 30920 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.553280 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:21.553777 30920 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.553735 30998 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.555812 31069 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:21.558354 31073 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.559072 30920 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.561446 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:21.562014 30926 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.567234 31039 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:21.567958 32199 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.568374 32199 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.569063 32199 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.570860 30926 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.572954 30998 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.572454 31073 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:21.575129 31039 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:21.575588 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:21.575708 32109 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.576220 30998 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.576583 30998 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:21.576515 32109 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.577582 30998 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.578004 32109 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.578964 30998 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.606336 30998 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.617599 30823 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.638321 30823 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.825659 32109 raft_consensus.cc:991] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.826141 32157 raft_consensus.cc:1079] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.827359 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:21.827872 31728 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.828219 31728 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.835126 31728 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.837410 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:21.837574 31728 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:21.838009 30926 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.837879 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:21.838696 31069 raft_consensus.cc:3053] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.838984 31069 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.839630 31069 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.840739 31069 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.843824 30926 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.844762 31693 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:21.845405 32217 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.846288 31069 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.846336 32217 raft_consensus.cc:695] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:21.847805 32217 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.865984 30823 catalog_manager.cc:5582] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.901984 32109 raft_consensus.cc:991] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.902418 32109 raft_consensus.cc:1079] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:21.903455 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:21.903906 31728 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.904196 31728 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.908515 31728 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.909953 31728 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:21.910501 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:21.911060 30920 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.911283 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:21.911909 31073 raft_consensus.cc:3053] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.912194 31073 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.913156 32058 raft_consensus.cc:991] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.912806 31073 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.913537 32058 raft_consensus.cc:1079] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:21.914014 31073 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.918207 30920 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.919107 31693 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:21.919720 31069 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:21.920177 31069 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:21.920459 31069 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.927791 32217 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.928431 32217 raft_consensus.cc:695] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:21.927587 31069 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.929445 31073 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:21.930441 32217 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:21.933332 31069 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:21.934424 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:21.934923 30997 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.933465 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:21.937539 30920 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:21.938287 30920 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:21.939126 30920 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.940533 30920 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:21.942880 30998 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:21.943603 30997 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.944164 32278 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:21.944533 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:21.945314 32109 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:21.945714 32109 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:21.946346 32109 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:21.947067 30920 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:21.960722 30823 catalog_manager.cc:5582] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:21.974153 31069 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:21.975423 32219 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:21.989926 30822 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:22.036979 30920 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.038074 32109 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.043567 30926 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.044983 32109 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.052834 30997 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.056280 32109 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.060770 31000 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.062155 32199 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.364866 31069 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:22.366070 32217 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:22.369027 31070 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:22.370576 32217 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:22.383824 30926 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:22.384882 32219 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:22.398039 30920 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:22.407281 32278 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:22.494367 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:22.501618 31000 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.504920 32157 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.508106 30926 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:22.516443 32157 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.533495 31000 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:22.598917 31073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:22.599472 31073 raft_consensus.cc:604] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:22.602348 31073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:22.602777 31073 raft_consensus.cc:604] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:22.607125 31073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2925593648194df5b7741d448a047401"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:22.607529 31073 raft_consensus.cc:604] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
W20250902 01:58:22.609359 30964 consensus_peers.cc:489] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39363bee7fbd45fdaa81366e5da31356. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:22.610316 30924 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:22.610857 30924 raft_consensus.cc:604] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:22.611815 30926 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.612730 31073 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.612972 32162 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.613734 30926 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:22.614078 30926 raft_consensus.cc:604] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:22.615507 32162 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.622702 31000 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:22.637619 31000 raft_consensus.cc:604] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:22.639264 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250902 01:58:22.640758 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:22.641602 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:22.659200 32162 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.663806 30926 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.664053 31073 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.679148 30924 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:22.680742 30823 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.681828 32109 raft_consensus.cc:991] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.682202 32109 raft_consensus.cc:1079] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.683599 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2925593648194df5b7741d448a047401"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:22.684021 31728 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.684304 31728 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.700999 31728 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.703876 31728 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:22.704748 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:22.705118 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:22.705205 31000 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.705631 31073 raft_consensus.cc:3053] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:22.705933 31073 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:22.706585 31073 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.707722 31073 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.712424 31000 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:22.713930 31692 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:22.714071 31073 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:22.714720 32217 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:22.716127 32217 raft_consensus.cc:695] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:22.717195 32217 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:22.742066 30889 consensus_peers.cc:489] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a3db1aaea8d4a3eba2de840361e0849. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:22.743575 31073 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.744764 32207 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.746359 32148 raft_consensus.cc:991] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:22.746757 32148 raft_consensus.cc:1079] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:22.748867 31073 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:22.749291 31073 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.749631 31073 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.744980 31000 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.755496 32207 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.755184 30824 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:22.778175 32058 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.775315 31073 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:22.784922 31070 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.788848 30997 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.790000 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:22.790652 31000 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.791682 31073 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:22.796348 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:22.796801 30924 raft_consensus.cc:3053] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:22.797173 30924 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:22.797923 30924 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:22.801331 30924 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.806162 31000 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:22.807446 31038 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:22.808519 32157 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:22.808903 32157 raft_consensus.cc:695] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:22.809572 32157 consensus_queue.cc:237] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:22.826504 30924 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:22.845747 30924 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:22.851114 30819 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:22.865381 30997 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.866259 32148 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:22.866947 31073 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:22.868551 32148 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250902 01:58:22.869196 30889 consensus_peers.cc:489] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24426e817ea94ac5a7014c3457ea087c. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:22.875355 30824 catalog_manager.cc:5582] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:22.879437 32058 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.892215 30997 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.890537 31073 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.904490 31000 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:22.916160 30822 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:22.925879 32201 raft_consensus.cc:991] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.926293 32201 raft_consensus.cc:1079] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
W20250902 01:58:22.931046 30964 consensus_peers.cc:489] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e12bca11fa644c12b4c6f0658fa07084. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:22.932925 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:22.933399 31728 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.933722 31728 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.934517 31073 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:22.936429 32162 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:22.939085 30920 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:22.941177 32213 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:22.941462 32148 raft_consensus.cc:991] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.943912 32148 raft_consensus.cc:1079] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.945209 31728 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.947057 32162 raft_consensus.cc:991] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:22.947530 32162 raft_consensus.cc:1079] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:22.948060 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:22.948565 30926 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.951655 31726 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51240
I20250902 01:58:22.952100 31726 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.952407 31726 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.966531 32157 raft_consensus.cc:991] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.967150 32157 raft_consensus.cc:1079] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:22.969985 30926 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:22.971101 31726 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.972873 31730 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:22.973338 31730 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.973625 31730 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:22.975430 31728 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:22.976199 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:22.979179 31070 raft_consensus.cc:3053] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:22.979471 31070 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:22.979521 30924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:22.980057 30924 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:22.980391 30924 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.980270 31070 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.981264 31070 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.984336 31730 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.977632 31693 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:22.978555 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:22.985455 30997 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.985678 30924 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:22.986202 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:22.986792 30926 raft_consensus.cc:3053] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:22.987082 30926 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:22.987830 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:22.987915 32217 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:22.988278 31069 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.988512 32217 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:22.989519 32217 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:22.994462 31069 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:22.995461 30892 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:22.996203 32058 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:22.996577 32058 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:22.997200 32058 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.002020 30924 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:23.004403 30997 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:23.008967 31692 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:23.006567 31070 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:23.010739 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:23.011250 30998 raft_consensus.cc:3053] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:23.011606 30998 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:23.011276 31726 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:23.011869 32219 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:23.013098 32219 raft_consensus.cc:695] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:23.013875 32219 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:23.012213 30998 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.016340 30998 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:22.987897 30926 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:23.018204 30926 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:23.030252 31730 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:23.032691 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:23.033179 31069 raft_consensus.cc:3053] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:23.033442 31069 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:23.034297 30998 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:23.034157 31069 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:23.040616 31069 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:23.075299 30926 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:23.080807 31069 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:23.079577 30824 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:23.087682 32324 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20250902 01:58:23.093590 31695 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:23.094703 32219 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:23.095096 32219 raft_consensus.cc:695] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:23.095832 32219 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:23.105893 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:23.106539 30924 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:23.153934 30824 catalog_manager.cc:5582] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:23.160080 32329 ts_tablet_manager.cc:927] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:23.159930 30924 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:23.161566 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 79353.8us
I20250902 01:58:23.169297 32327 ts_tablet_manager.cc:927] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:23.173211 30819 catalog_manager.cc:5582] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:23.175020 30819 catalog_manager.cc:5582] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 2 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:23.177433 32213 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:23.191155 32329 tablet_copy_client.cc:323] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:23.192224 31073 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.192286 32327 tablet_copy_client.cc:323] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:23.193712 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet 39363bee7fbd45fdaa81366e5da31356 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:23.193662 32217 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.194232 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet 39363bee7fbd45fdaa81366e5da31356 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-39363bee7fbd45fdaa81366e5da31356
I20250902 01:58:23.196951 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 5a3db1aaea8d4a3eba2de840361e0849 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:23.197439 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 5a3db1aaea8d4a3eba2de840361e0849 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-5a3db1aaea8d4a3eba2de840361e0849
I20250902 01:58:23.203125 30924 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:23.196532 31072 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:23.209528 30920 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:23.210788 31011 tablet_copy_source_session.cc:215] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:23.223433 32201 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:23.227034 30819 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:23.238636 30936 tablet_copy_source_session.cc:215] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:23.240129 30998 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.241480 32219 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.248267 32329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39363bee7fbd45fdaa81366e5da31356. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:23.254251 32327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3db1aaea8d4a3eba2de840361e0849. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:23.307302 32329 tablet_copy_client.cc:806] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:23.307848 32329 tablet_copy_client.cc:670] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:23.308964 30998 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:23.310827 32109 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:23.318984 32329 tablet_copy_client.cc:538] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:23.331079 32327 tablet_copy_client.cc:806] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:23.331763 32327 tablet_copy_client.cc:670] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:23.376355 32327 tablet_copy_client.cc:538] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:23.380971 32329 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:23.389355 32327 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:23.420738 32327 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:23.421629 32327 tablet_bootstrap.cc:492] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:23.422235 32327 ts_tablet_manager.cc:1397] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.033s user 0.018s sys 0.008s
I20250902 01:58:23.424767 32327 raft_consensus.cc:357] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.425532 32327 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:23.426481 32343 ts_tablet_manager.cc:927] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:23.426128 32327 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.428046 32327 ts_tablet_manager.cc:1428] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250902 01:58:23.433624 31070 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.436880 32217 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.438179 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-5a3db1aaea8d4a3eba2de840361e0849 received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:23.438644 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-5a3db1aaea8d4a3eba2de840361e0849 on tablet 5a3db1aaea8d4a3eba2de840361e0849 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:23.445215 32329 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:23.446403 32329 tablet_bootstrap.cc:492] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:23.446874 32329 ts_tablet_manager.cc:1397] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.066s user 0.015s sys 0.006s
I20250902 01:58:23.455348 32329 raft_consensus.cc:357] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.456032 32329 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:23.456506 32329 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.458737 32329 ts_tablet_manager.cc:1428] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250902 01:58:23.464138 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-39363bee7fbd45fdaa81366e5da31356 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:23.464532 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-39363bee7fbd45fdaa81366e5da31356 on tablet 39363bee7fbd45fdaa81366e5da31356 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:23.481765 30920 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.485422 31730 raft_consensus.cc:1215] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:23.487506 32321 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.489362 32343 tablet_copy_client.cc:323] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:23.490491 32329 ts_tablet_manager.cc:927] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:23.496145 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet e12bca11fa644c12b4c6f0658fa07084 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:23.496624 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet e12bca11fa644c12b4c6f0658fa07084 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-e12bca11fa644c12b4c6f0658fa07084
I20250902 01:58:23.497169 30997 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:23.502120 32058 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:23.503314 31011 tablet_copy_source_session.cc:215] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:23.507705 32343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e12bca11fa644c12b4c6f0658fa07084. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:23.540226 31070 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:23.540992 32343 tablet_copy_client.cc:806] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:23.541296 32313 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:23.541505 32343 tablet_copy_client.cc:670] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:23.545629 31070 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:23.546759 32217 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.550320 32343 tablet_copy_client.cc:538] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:23.553988 32329 tablet_copy_client.cc:323] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:23.555925 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 24426e817ea94ac5a7014c3457ea087c from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:23.556386 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 24426e817ea94ac5a7014c3457ea087c from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-24426e817ea94ac5a7014c3457ea087c
I20250902 01:58:23.569123 30936 tablet_copy_source_session.cc:215] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:23.572027 32329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24426e817ea94ac5a7014c3457ea087c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:23.573169 32343 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:23.591665 30920 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:23.592983 32278 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.597126 32329 tablet_copy_client.cc:806] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:23.597594 32329 tablet_copy_client.cc:670] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:23.604144 31726 raft_consensus.cc:1215] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:23.610572 30997 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.611243 32329 tablet_copy_client.cc:538] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:23.611948 32278 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.626531 30926 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:23.627803 32217 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:23.646178 32329 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:23.732594 32343 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:23.733721 32343 tablet_bootstrap.cc:492] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:23.734393 32343 ts_tablet_manager.cc:1397] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.161s user 0.018s sys 0.015s
I20250902 01:58:23.738327 32343 raft_consensus.cc:357] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.739948 32343 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:23.740689 32343 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.743969 32343 ts_tablet_manager.cc:1428] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250902 01:58:23.747128 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-e12bca11fa644c12b4c6f0658fa07084 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:23.747644 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-e12bca11fa644c12b4c6f0658fa07084 on tablet e12bca11fa644c12b4c6f0658fa07084 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:23.757969 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:23.818249 32329 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:23.819271 32329 tablet_bootstrap.cc:492] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:23.820001 32329 ts_tablet_manager.cc:1397] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.174s user 0.059s sys 0.081s
I20250902 01:58:23.822538 32329 raft_consensus.cc:357] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.823294 32329 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:23.825747 32329 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:23.828249 32329 ts_tablet_manager.cc:1428] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250902 01:58:23.831279 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-24426e817ea94ac5a7014c3457ea087c received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:23.831714 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-24426e817ea94ac5a7014c3457ea087c on tablet 24426e817ea94ac5a7014c3457ea087c with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:23.836355 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:23.836779 30920 raft_consensus.cc:604] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:23.838841 30997 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:23.839288 30997 raft_consensus.cc:604] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:23.840610 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250902 01:58:23.841893 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:23.842494 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:23.868453 31726 raft_consensus.cc:1215] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250902 01:58:23.878710 32332 raft_consensus.cc:1062] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:23.900710 31730 raft_consensus.cc:1215] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:23.895310 32207 raft_consensus.cc:991] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:23.902400 32207 raft_consensus.cc:1079] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:23.905534 31070 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:23.905983 31070 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:23.906303 31070 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:23.911391 32332 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:23.926447 32162 raft_consensus.cc:991] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:23.926859 32162 raft_consensus.cc:1079] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:23.928071 30920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:23.928514 30920 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:23.928828 30920 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:23.933189 31070 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.939277 30920 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.944371 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:23.944839 30926 raft_consensus.cc:3053] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:23.945057 30926 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:23.945389 30920 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:23.945643 30926 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.946579 30926 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:23.946679 31070 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:23.948246 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:23.948761 30997 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:23.950261 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:23.950829 30998 raft_consensus.cc:3053] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:23.951129 30998 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:23.951824 30998 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.953220 30998 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:23.959002 30926 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:23.959852 30998 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:23.964213 31726 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:23.964614 31039 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:23.965272 32109 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:23.965628 32109 raft_consensus.cc:695] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:23.966159 32109 consensus_queue.cc:237] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.968535 32162 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:23.963845 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:23.970096 31070 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:23.974944 31072 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:23.976544 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:23.977268 32313 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:23.977638 32313 raft_consensus.cc:695] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:23.978329 32313 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:23.982887 32213 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:23.988732 31070 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:24.002897 30824 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:23.996918 30819 catalog_manager.cc:5582] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:23.960414 30997 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:24.021140 32162 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.024653 31730 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.038534 30926 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.038185 31070 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.041321 32359 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.052862 30926 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.058967 30824 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.072713 32357 raft_consensus.cc:1062] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:24.074928 32357 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.088402 31726 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.089232 31070 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.089488 32148 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.091003 32148 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.094005 30998 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.095992 32313 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250902 01:58:24.111260 32313 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.113603 31000 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.118257 31726 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.128571 30823 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.131556 31070 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.145078 30926 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.153535 31726 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.154587 32313 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.155892 31070 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.159637 32313 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.160827 32148 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.167869 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5a3db1aaea8d4a3eba2de840361e0849 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:24.170710 31726 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.179436 31070 raft_consensus.cc:2953] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.183000 30822 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:24.198427 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a3db1aaea8d4a3eba2de840361e0849 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:24.208703 32372 tablet_replica.cc:331] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:24.209849 32372 raft_consensus.cc:2241] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:24.210371 32372 raft_consensus.cc:2270] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:24.213451 32372 ts_tablet_manager.cc:1905] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:24.227149 32372 ts_tablet_manager.cc:1918] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:24.227523 32372 log.cc:1199] T 5a3db1aaea8d4a3eba2de840361e0849 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/5a3db1aaea8d4a3eba2de840361e0849
I20250902 01:58:24.229143 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 5a3db1aaea8d4a3eba2de840361e0849 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:24.345499 30926 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:24.347802 31000 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:24.349514 32157 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:24.350170 32109 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:24.376426 31001 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:24.377516 32313 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.405042 31070 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:24.410190 32357 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.426680 32357 raft_consensus.cc:1062] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:24.428857 32357 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.440269 32213 raft_consensus.cc:1062] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:24.442211 32213 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.459446 31726 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:24.460958 32359 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.471716 30926 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:24.472802 31070 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:24.473232 32359 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.474027 32162 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.477399 31730 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.483314 32207 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.489027 31000 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.500483 31073 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:24.501375 32207 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.509109 32058 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:24.529049 32213 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.536739 31730 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.539096 31073 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.547331 30822 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:24.562611 32207 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.570142 31000 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.599715 31730 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:24.601027 32162 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:58:24.602434 31070 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:24.604727 32213 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:24.614626 31000 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.623780 32213 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.632670 31073 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.640763 31730 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.665387 30823 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:24.674204 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51254us
I20250902 01:58:24.677399 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e12bca11fa644c12b4c6f0658fa07084 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:24.686291 30819 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:24.705765 30920 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.713544 31730 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.718577 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet e12bca11fa644c12b4c6f0658fa07084 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:24.720824 31070 raft_consensus.cc:2953] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.731276 31726 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.737700 31001 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.739835 32148 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.740671 32395 tablet_replica.cc:331] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:24.741299 32148 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.741649 32395 raft_consensus.cc:2241] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:24.742197 32395 pending_rounds.cc:70] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Trying to abort 1 pending ops.
I20250902 01:58:24.742455 32395 pending_rounds.cc:77] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Aborting op as it isn't in flight: id { term: 2 index: 4 } timestamp: 7195763934990700544 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e12bca11fa644c12b4c6f0658fa07084" old_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } } new_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } } }
I20250902 01:58:24.758250 32395 raft_consensus.cc:2915] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Aborting config change with OpId 2.4: Aborted: Op aborted
I20250902 01:58:24.757550 32313 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.758666 32395 raft_consensus.cc:2887] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250902 01:58:24.759042 32395 raft_consensus.cc:2270] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:24.761927 32395 ts_tablet_manager.cc:1905] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:24.772540 31730 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.796437 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 24426e817ea94ac5a7014c3457ea087c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:24.802060 30998 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:24.808025 31726 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.810986 32213 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.818800 31000 raft_consensus.cc:2953] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.808027 30926 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:24.821511 32359 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:24.824098 32359 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.828040 30823 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.831015 31726 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.836220 32395 ts_tablet_manager.cc:1918] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250902 01:58:24.838132 32395 log.cc:1199] T e12bca11fa644c12b4c6f0658fa07084 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/e12bca11fa644c12b4c6f0658fa07084
I20250902 01:58:24.840206 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet e12bca11fa644c12b4c6f0658fa07084 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:24.844729 30920 raft_consensus.cc:2953] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.846336 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 39363bee7fbd45fdaa81366e5da31356 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:24.857875 30823 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:24.867245 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 24426e817ea94ac5a7014c3457ea087c with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:24.876981 32396 tablet_replica.cc:331] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:24.877885 32396 raft_consensus.cc:2241] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:24.878521 32396 raft_consensus.cc:2270] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:24.882777 32396 ts_tablet_manager.cc:1905] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:24.885504 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 39363bee7fbd45fdaa81366e5da31356 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:24.892494 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:24.909472 32396 ts_tablet_manager.cc:1918] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:24.909969 32396 log.cc:1199] T 24426e817ea94ac5a7014c3457ea087c P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/24426e817ea94ac5a7014c3457ea087c
I20250902 01:58:24.911433 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 24426e817ea94ac5a7014c3457ea087c (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:24.911875 32396 tablet_replica.cc:331] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:24.913342 32396 raft_consensus.cc:2241] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:24.914000 32396 raft_consensus.cc:2270] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:24.917157 32396 ts_tablet_manager.cc:1905] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:24.938066 32396 ts_tablet_manager.cc:1918] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:24.938454 32396 log.cc:1199] T 39363bee7fbd45fdaa81366e5da31356 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/39363bee7fbd45fdaa81366e5da31356
I20250902 01:58:24.939864 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 39363bee7fbd45fdaa81366e5da31356 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:24.940135 31001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "349593d2d3574cd6ab281c105756f435"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:24.940616 31001 raft_consensus.cc:604] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:24.943333 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:24.943747 31069 raft_consensus.cc:604] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:24.945677 30926 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:24.946079 30926 raft_consensus.cc:604] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:24.948117 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:24.948450 31069 raft_consensus.cc:604] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:24.949218 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250902 01:58:24.950711 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:24.951339 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:25.041054 32359 raft_consensus.cc:991] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:25.041440 32393 raft_consensus.cc:1079] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:25.042706 30920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "349593d2d3574cd6ab281c105756f435"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:25.043227 30920 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:25.043588 30920 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:25.050186 30920 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.052165 30920 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:25.053114 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:25.053548 31001 raft_consensus.cc:3053] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:25.053804 31001 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:25.054287 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:25.054857 31730 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:25.055071 31001 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.056160 31001 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:25.060868 31730 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:25.062790 31001 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:25.064038 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:25.064658 32148 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:25.065014 32148 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:25.065639 32148 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.077876 30822 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:25.187880 32207 raft_consensus.cc:991] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:25.188344 32207 raft_consensus.cc:1079] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:25.189746 31069 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:25.190230 31069 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:25.190534 31069 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.194609 31069 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.196298 31069 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:25.196421 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:25.197090 30920 raft_consensus.cc:3053] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:25.197365 30920 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:25.197757 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:25.198221 31001 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.198012 30920 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.199220 30920 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.203737 31001 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:25.205111 31038 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:25.205530 30920 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:25.205801 32392 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:25.206300 32392 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:25.207113 32392 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.219177 30822 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:25.244441 32201 raft_consensus.cc:991] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:25.244793 32392 raft_consensus.cc:1079] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:25.245815 31730 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:25.246132 31730 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:25.246330 31730 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:25.251084 31730 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.253085 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:25.253407 31730 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:25.253575 31001 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:25.257334 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:25.257977 31069 raft_consensus.cc:3053] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:25.258229 31069 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:25.259234 31069 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.260643 31069 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:25.261296 31001 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:25.262271 31692 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:25.262902 32278 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:25.263284 32278 raft_consensus.cc:695] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:25.264070 32278 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:25.271350 31069 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:25.274293 30822 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:25.335371 32201 raft_consensus.cc:991] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:25.335780 32392 raft_consensus.cc:1079] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:25.337576 31001 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:25.338063 31001 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:25.338382 31001 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.342974 31001 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.344666 31001 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:25.345885 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:25.346439 30920 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.346437 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:25.347170 31070 raft_consensus.cc:3053] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:25.347553 31070 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:25.348148 31070 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.349208 31070 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:25.352334 30920 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:25.353371 30965 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:25.354028 32393 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:25.354400 32393 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:25.355074 32393 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:25.368450 31070 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:25.369056 30822 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:25.571120 31726 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:25.572225 32207 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:25.593994 31001 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:25.598510 32357 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:25.616535 31001 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:25.618701 32201 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:25.653123 30926 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:25.654477 32392 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:25.681259 31070 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:25.682633 32219 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:25.689338 30998 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:25.691135 32278 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:25.747053 31070 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:25.748260 32393 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:25.755450 30926 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:25.757745 32213 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:25.934521 30920 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:25.940562 31001 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:25.939455 31070 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:25.941905 32207 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:25.942613 32148 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250902 01:58:25.943279 30889 consensus_peers.cc:489] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 151f832cfd4e42d3a547a7cef6bb995f. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:25.954231 32357 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:25.955076 31070 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:25.961753 31001 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:25.970067 30822 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:25.971452 30998 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:25.978314 30964 consensus_peers.cc:489] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ade302bdd69a4fa7876c90a1fbd87b68. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:25.978339 31070 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:25.984474 30920 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:25.987571 32393 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:25.988934 32393 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:25.996285 32359 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:25.997644 31069 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.000838 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:26.002090 30926 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.017827 31069 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:26.029362 31038 consensus_peers.cc:489] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 416020230983423c9f843070a7893658. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:26.030161 30998 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:26.031735 32392 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:26.032471 30926 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:26.034155 32392 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:26.058230 31001 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.067116 31069 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:26.067590 31069 raft_consensus.cc:604] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:26.067597 30822 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.079751 30920 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.056362 32392 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.090220 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:26.090744 30920 raft_consensus.cc:604] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:26.094261 30998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:26.094697 30998 raft_consensus.cc:604] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:26.097200 31070 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.104328 30819 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
W20250902 01:58:26.110412 31038 consensus_peers.cc:489] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de2e42094ddd41f99d12a4972ee07b9f. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:26.110632 30926 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:26.113782 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250902 01:58:26.114130 30998 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:26.115969 32406 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:26.115371 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:26.122084 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:26.126300 32392 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.141968 32157 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:26.147465 30998 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.160055 32201 raft_consensus.cc:991] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:26.160511 32406 raft_consensus.cc:1079] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:26.165124 30822 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:26.175603 30924 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:26.163110 31001 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:26.178046 31001 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:26.178364 31001 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.184274 31001 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.185722 31001 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:26.186671 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:26.187575 31726 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.187466 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:26.188153 31069 raft_consensus.cc:3053] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:26.188398 31069 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:26.189098 31069 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.190213 31069 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.193982 31726 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:26.195011 30964 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:26.195951 32413 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:26.196336 32413 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:26.197012 32413 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.289251 31069 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:26.363248 32359 raft_consensus.cc:991] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:26.364022 32213 raft_consensus.cc:1079] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:26.368665 30924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:26.369618 30924 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:26.370110 30924 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.368927 30823 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:26.383893 30924 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.418607 30924 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:26.420140 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:26.420960 31730 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.422201 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:26.422832 30998 raft_consensus.cc:3053] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:26.423151 30998 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:26.424022 30998 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.425592 30998 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.448352 30998 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:26.449769 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:26.450249 32431 ts_tablet_manager.cc:927] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:26.450800 32148 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:26.451402 32148 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:26.452375 32148 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:26.466161 31730 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:26.490922 30823 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:26.508574 32430 ts_tablet_manager.cc:927] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:26.530148 32431 tablet_copy_client.cc:323] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:26.532388 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet ade302bdd69a4fa7876c90a1fbd87b68 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:26.533156 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet ade302bdd69a4fa7876c90a1fbd87b68 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-ade302bdd69a4fa7876c90a1fbd87b68
I20250902 01:58:26.543740 32430 tablet_copy_client.cc:323] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:26.545614 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 151f832cfd4e42d3a547a7cef6bb995f from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:26.546139 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 151f832cfd4e42d3a547a7cef6bb995f from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-151f832cfd4e42d3a547a7cef6bb995f
I20250902 01:58:26.551661 31011 tablet_copy_source_session.cc:215] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:26.553939 30936 tablet_copy_source_session.cc:215] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:26.557458 32430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151f832cfd4e42d3a547a7cef6bb995f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:26.558979 32431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade302bdd69a4fa7876c90a1fbd87b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:26.574740 32430 tablet_copy_client.cc:806] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:26.575322 32430 tablet_copy_client.cc:670] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:26.581001 32148 raft_consensus.cc:991] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:26.581415 32148 raft_consensus.cc:1079] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:26.581537 32431 tablet_copy_client.cc:806] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:26.582176 32431 tablet_copy_client.cc:670] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:26.582934 32430 tablet_copy_client.cc:538] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:26.583462 31073 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:26.583958 31073 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:26.584254 31073 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.586863 32431 tablet_copy_client.cc:538] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:26.590274 31073 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:26.594903 32430 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:26.596678 31073 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:26.598165 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:26.598289 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:26.598796 31000 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.598876 30926 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:26.599287 30926 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:26.600214 30926 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:26.601507 30926 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:26.605453 31000 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:26.606217 32431 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:26.607045 31038 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:26.608055 32406 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:26.608450 32406 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:26.608768 30926 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:26.609223 32406 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:26.623157 32436 ts_tablet_manager.cc:927] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:26.627470 32436 tablet_copy_client.cc:323] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:26.645393 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 416020230983423c9f843070a7893658 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:26.645951 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 416020230983423c9f843070a7893658 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-416020230983423c9f843070a7893658
I20250902 01:58:26.647039 30823 catalog_manager.cc:5582] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:26.650696 32430 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:26.652209 32430 tablet_bootstrap.cc:492] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:26.653003 32430 ts_tablet_manager.cc:1397] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.058s user 0.026s sys 0.005s
I20250902 01:58:26.656373 31084 tablet_copy_source_session.cc:215] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:26.659675 32436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416020230983423c9f843070a7893658. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:26.664342 32430 raft_consensus.cc:357] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.665100 32430 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:26.665689 32430 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.671845 32430 ts_tablet_manager.cc:1428] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20250902 01:58:26.676771 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-151f832cfd4e42d3a547a7cef6bb995f received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:26.677223 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-151f832cfd4e42d3a547a7cef6bb995f on tablet 151f832cfd4e42d3a547a7cef6bb995f with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:26.677451 32436 tablet_copy_client.cc:806] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:26.677973 32436 tablet_copy_client.cc:670] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:26.680095 32431 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:26.683620 32431 tablet_bootstrap.cc:492] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:26.684732 32436 tablet_copy_client.cc:538] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:26.687774 32431 ts_tablet_manager.cc:1397] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.082s user 0.027s sys 0.003s
I20250902 01:58:26.690510 32431 raft_consensus.cc:357] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.691222 32431 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:26.698828 32436 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:26.691859 32431 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.712066 32431 ts_tablet_manager.cc:1428] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20250902 01:58:26.715219 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-ade302bdd69a4fa7876c90a1fbd87b68 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:26.715633 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-ade302bdd69a4fa7876c90a1fbd87b68 on tablet ade302bdd69a4fa7876c90a1fbd87b68 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:26.729326 32431 ts_tablet_manager.cc:927] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:26.736553 32431 tablet_copy_client.cc:323] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:26.741487 32436 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:26.742197 32436 tablet_bootstrap.cc:492] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:26.742664 32436 ts_tablet_manager.cc:1397] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.004s
I20250902 01:58:26.743561 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet de2e42094ddd41f99d12a4972ee07b9f from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:26.744095 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet de2e42094ddd41f99d12a4972ee07b9f from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-de2e42094ddd41f99d12a4972ee07b9f
I20250902 01:58:26.745097 32436 raft_consensus.cc:357] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.745793 32436 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:26.746312 32436 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.748874 32436 ts_tablet_manager.cc:1428] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250902 01:58:26.763769 31084 tablet_copy_source_session.cc:215] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:26.765504 31730 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:26.767066 32429 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:26.768438 32431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de2e42094ddd41f99d12a4972ee07b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:26.777825 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-416020230983423c9f843070a7893658 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:26.778261 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-416020230983423c9f843070a7893658 on tablet 416020230983423c9f843070a7893658 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:26.780855 31070 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:26.782866 32359 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:26.808012 32431 tablet_copy_client.cc:806] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:26.808552 32431 tablet_copy_client.cc:670] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:26.828842 31728 raft_consensus.cc:1215] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20250902 01:58:26.851408 32431 tablet_copy_client.cc:538] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:26.859158 32431 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:26.881072 32431 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:26.881860 32431 tablet_bootstrap.cc:492] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:26.882428 32431 ts_tablet_manager.cc:1397] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.007s
I20250902 01:58:26.884343 32431 raft_consensus.cc:357] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.884933 32431 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:26.885346 32431 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:26.887377 32431 ts_tablet_manager.cc:1428] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250902 01:58:26.891245 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-de2e42094ddd41f99d12a4972ee07b9f received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:26.891615 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-de2e42094ddd41f99d12a4972ee07b9f on tablet de2e42094ddd41f99d12a4972ee07b9f with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:26.905496 31001 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:26.907123 32207 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:26.937436 31730 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:26.938598 32207 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.005275 31000 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:27.006608 32392 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:27.009984 31730 raft_consensus.cc:1215] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:27.018354 30920 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:27.020749 32392 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:27.095485 31730 raft_consensus.cc:1215] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250902 01:58:27.228926 32429 raft_consensus.cc:1062] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:27.231045 32429 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.236523 31730 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.236792 31070 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.237668 32359 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.238018 30926 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.239279 32213 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.240048 32429 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.251195 32359 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.252897 30926 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.259886 31073 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.268069 30822 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.273640 31730 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.297477 31001 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.304550 30926 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:27.305938 32359 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:27.307004 31726 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:27.309398 32213 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:27.315173 32413 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.317121 30926 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.327663 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ade302bdd69a4fa7876c90a1fbd87b68 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:27.332880 31730 raft_consensus.cc:2953] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.336386 30824 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 5 to 6, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:27.348259 31726 raft_consensus.cc:1215] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20250902 01:58:27.357904 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet ade302bdd69a4fa7876c90a1fbd87b68 with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:27.360414 32461 tablet_replica.cc:331] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:27.361497 32461 raft_consensus.cc:2241] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:27.362139 32461 raft_consensus.cc:2270] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:27.367004 32461 ts_tablet_manager.cc:1905] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:27.381703 32461 ts_tablet_manager.cc:1918] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250902 01:58:27.382104 32461 log.cc:1199] T ade302bdd69a4fa7876c90a1fbd87b68 P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/ade302bdd69a4fa7876c90a1fbd87b68
I20250902 01:58:27.383522 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet ade302bdd69a4fa7876c90a1fbd87b68 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:27.422956 32207 raft_consensus.cc:1062] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:27.425071 32207 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.433593 31726 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:27.434024 31073 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:27.434623 32425 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:27.435432 31001 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:27.435729 32207 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:27.437481 32425 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:27.448446 32148 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.452322 31726 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.452621 31001 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.458093 31070 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.466303 30823 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:27.479995 30926 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.486964 31070 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:27.488950 32148 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:27.498262 32425 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.499982 31069 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.502658 31726 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:27.505646 32148 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:27.507860 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:27.509816 30809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 151f832cfd4e42d3a547a7cef6bb995f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:27.510943 31726 raft_consensus.cc:2953] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.516381 30823 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.527065 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 151f832cfd4e42d3a547a7cef6bb995f with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:27.539413 32466 tablet_replica.cc:331] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:27.548597 32466 raft_consensus.cc:2241] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:27.549417 32466 raft_consensus.cc:2270] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:27.552731 32466 ts_tablet_manager.cc:1905] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:27.573434 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:27.573979 30920 raft_consensus.cc:604] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:27.576412 32466 ts_tablet_manager.cc:1918] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:27.576808 32466 log.cc:1199] T 151f832cfd4e42d3a547a7cef6bb995f P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/151f832cfd4e42d3a547a7cef6bb995f
I20250902 01:58:27.577646 31070 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:27.578204 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 151f832cfd4e42d3a547a7cef6bb995f (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:27.578151 31070 raft_consensus.cc:604] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:27.581164 31001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:27.581569 31001 raft_consensus.cc:604] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:27.582554 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250902 01:58:27.583909 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:27.584745 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:27.587692 32406 raft_consensus.cc:991] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:27.588166 32427 raft_consensus.cc:1079] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:27.589360 31001 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53758
I20250902 01:58:27.589814 31001 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:27.590134 31001 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:27.595233 31001 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:27.597255 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:27.597877 30920 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:27.602725 31001 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:27.602886 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:27.603924 30920 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:27.639884 31069 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:27.642227 31069 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:27.643308 31069 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:27.648955 31069 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:27.655900 30965 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:27.657232 32359 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:27.657683 32359 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:58:27.657574 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 49338.1us
I20250902 01:58:27.658617 32359 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:27.660846 31069 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:27.672834 32427 raft_consensus.cc:1062] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:27.675040 32427 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.684976 31726 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:27.684996 30920 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:27.684613 30819 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 4 to 5, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:27.686179 31001 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:27.686417 32427 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:27.687259 32392 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250902 01:58:27.690671 32406 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:27.707353 32406 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.708652 31730 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.711074 31001 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.724459 30819 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.727442 32425 raft_consensus.cc:991] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:27.727866 32425 raft_consensus.cc:1079] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:27.736712 30920 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.744202 31070 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:27.744738 31070 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:27.745095 31070 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:27.747153 31069 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.758591 31726 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:27.760507 31001 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:27.760413 31070 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.760848 32427 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.762104 31070 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:27.762641 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:27.763163 30924 raft_consensus.cc:3053] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:27.763397 30924 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:27.764000 30924 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.765053 30924 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:27.765869 32427 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.766034 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:27.766522 31726 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:27.769102 32392 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.770867 30924 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:27.773653 31039 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:27.774612 31726 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:27.774701 32406 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:27.775804 32406 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:27.776671 32406 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.778739 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 416020230983423c9f843070a7893658 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:27.778120 31726 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.788424 31001 raft_consensus.cc:2953] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.794859 30819 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.802575 30824 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:27.805932 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 416020230983423c9f843070a7893658 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:27.808759 32474 tablet_replica.cc:331] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:27.809749 32474 raft_consensus.cc:2241] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:27.810492 32474 raft_consensus.cc:2270] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:27.813401 32474 ts_tablet_manager.cc:1905] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:27.827216 32474 ts_tablet_manager.cc:1918] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250902 01:58:27.827584 32474 log.cc:1199] T 416020230983423c9f843070a7893658 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/416020230983423c9f843070a7893658
I20250902 01:58:27.829052 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 416020230983423c9f843070a7893658 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:27.957080 32472 raft_consensus.cc:1062] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:27.959553 32472 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:27.964277 31726 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.964825 30920 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.965819 31001 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:27.965849 32472 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250902 01:58:27.968309 32406 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.969684 32392 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:27.982056 32406 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.983574 31726 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.997202 32413 raft_consensus.cc:991] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:27.996697 30998 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:27.997634 32359 raft_consensus.cc:1079] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:27.999153 30819 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:28.002173 30924 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:28.002885 30924 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:28.003269 30924 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.008760 30920 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:28.007782 30924 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.023463 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:28.023805 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:28.024094 30998 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:28.024276 31726 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.024452 30998 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:28.025105 30924 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:28.025223 30998 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.026459 30998 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.031750 31726 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:28.034585 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:28.034817 31070 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.035203 32207 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:28.035745 32207 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:28.036857 32207 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.037942 30998 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:28.041378 30924 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:28.043710 32427 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:28.044533 31726 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:28.050460 32427 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:28.053630 32392 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:28.054777 30924 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:28.062867 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet de2e42094ddd41f99d12a4972ee07b9f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:28.062863 30819 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:28.071892 31726 raft_consensus.cc:2953] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:28.072921 30824 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 5 to 6, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:28.081511 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet de2e42094ddd41f99d12a4972ee07b9f with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:28.082979 32481 tablet_replica.cc:331] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:28.084707 32481 raft_consensus.cc:2241] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:28.086956 32481 raft_consensus.cc:2270] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:28.090896 32481 ts_tablet_manager.cc:1905] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:28.104609 32481 ts_tablet_manager.cc:1918] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250902 01:58:28.104943 32481 log.cc:1199] T de2e42094ddd41f99d12a4972ee07b9f P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/de2e42094ddd41f99d12a4972ee07b9f
I20250902 01:58:28.106043 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet de2e42094ddd41f99d12a4972ee07b9f (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:28.170790 30924 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:28.171866 32413 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:28.180819 31070 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:28.183662 32213 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:28.207921 30920 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:28.208923 32427 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:28.219305 31726 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:28.221856 32406 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:28.516425 30998 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:28.517431 32148 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:28.524402 31726 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:28.525818 32494 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:28.664166 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:28.694087 30920 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:28.694658 30920 raft_consensus.cc:604] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:28.697122 31068 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8f5d81000607410da8880ef8e5517680"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:28.697609 31068 raft_consensus.cc:604] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:28.699853 30998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:48184
I20250902 01:58:28.700276 30998 raft_consensus.cc:604] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:28.701054 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250902 01:58:28.701941 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:28.702378 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:28.822773 32148 raft_consensus.cc:991] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:28.823100 32207 raft_consensus.cc:1079] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 088424002a894fd5bcc11a06af111445 to start an election
I20250902 01:58:28.824054 31068 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14"
dest_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:46556
I20250902 01:58:28.824486 31068 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:28.824765 31068 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.829082 31068 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.830216 31068 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:28.830809 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:28.831034 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:28.831324 30920 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:28.831648 30920 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:28.831506 31728 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.832298 30920 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.833154 30920 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:28.836745 31728 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:28.838127 30920 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:28.838285 31038 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:28.839147 32427 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:28.839608 32427 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:28.840399 32427 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:28.853777 30823 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:29.053874 32213 raft_consensus.cc:991] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:29.054282 32413 raft_consensus.cc:1079] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:29.055567 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:56978
I20250902 01:58:29.056039 31728 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:29.056349 31728 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:29.061625 31728 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.062996 31728 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:29.063606 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:29.063795 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:29.064153 31001 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:29.064314 31068 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:29.064421 31001 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:29.065042 31001 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.065821 31001 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:29.067982 31068 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:29.068895 31695 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:29.069504 32479 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:29.069809 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:29.069994 32479 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:29.070719 32479 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.080874 30823 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 2 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:29.129716 31001 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.133706 31070 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250902 01:58:29.133880 30964 consensus_peers.cc:489] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 805e9f7636dc47059b5af8c090e5ac0a. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:29.133899 30920 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250902 01:58:29.134591 32213 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.135600 32413 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.142092 30920 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.142004 31070 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.142997 32413 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.155100 30823 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 5, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.161799 31001 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.167182 31068 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:29.167227 30924 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:29.167683 30964 consensus_peers.cc:489] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa4b903a93464946ae238ce634b483d0. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:29.168455 32496 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:29.169945 32213 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:29.180491 32413 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.185629 30924 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.193394 31068 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.196099 31072 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.199581 30998 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250902 01:58:29.199676 31038 consensus_peers.cc:489] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48e0a90ceb7a486aa3a93a73f087dfc6. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:29.200151 30920 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:29.201295 32489 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:29.198544 30819 catalog_manager.cc:5582] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:29.202809 32489 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:29.209470 32489 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.211478 30998 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.223241 31072 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:29.228389 31038 consensus_peers.cc:489] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c9675a8370e4d4a9f95b3c568abe972. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:29.230140 31001 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:29.231122 32490 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:29.231374 30924 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250902 01:58:29.233265 32490 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:29.243398 30920 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.246812 30824 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:29.271111 32490 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.271899 31001 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.284644 30823 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.286803 30924 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:29.293015 31728 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:29.296550 32490 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.298686 32427 raft_consensus.cc:991] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:29.299391 32427 raft_consensus.cc:1079] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:29.301739 31728 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8f5d81000607410da8880ef8e5517680"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:58:29.302201 31728 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:29.302496 31728 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:29.307102 31728 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.309953 31728 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:29.310703 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:29.311262 31001 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:29.311661 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:29.312326 31072 raft_consensus.cc:3053] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:29.312660 31072 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:29.313493 31072 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.314862 31072 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:29.315161 31001 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:29.324595 31692 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:29.325383 32219 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:29.325769 32219 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:29.326300 32219 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:29.327191 31072 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:29.334616 30924 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:29.337579 32512 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.351824 30819 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:29.607201 31001 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:29.608146 32219 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:29.615412 31070 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:29.616145 32520 ts_tablet_manager.cc:927] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:29.617411 32219 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:29.646099 32520 tablet_copy_client.cc:323] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:29.647892 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:29.648540 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 48e0a90ceb7a486aa3a93a73f087dfc6 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-48e0a90ceb7a486aa3a93a73f087dfc6
I20250902 01:58:29.657526 32525 ts_tablet_manager.cc:927] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:29.669507 32525 tablet_copy_client.cc:323] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:29.670924 31084 tablet_copy_source_session.cc:215] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:29.671389 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet 805e9f7636dc47059b5af8c090e5ac0a from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:29.672106 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet 805e9f7636dc47059b5af8c090e5ac0a from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-805e9f7636dc47059b5af8c090e5ac0a
I20250902 01:58:29.681269 32520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e0a90ceb7a486aa3a93a73f087dfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:29.692524 31011 tablet_copy_source_session.cc:215] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:29.697196 32525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805e9f7636dc47059b5af8c090e5ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:29.698765 32520 tablet_copy_client.cc:806] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:29.699265 32520 tablet_copy_client.cc:670] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:29.702081 32527 ts_tablet_manager.cc:927] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:29.702900 32520 tablet_copy_client.cc:538] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:29.711211 32520 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:29.711683 32525 tablet_copy_client.cc:806] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:29.712275 32525 tablet_copy_client.cc:670] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:29.716101 32525 tablet_copy_client.cc:538] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:29.724555 32527 tablet_copy_client.cc:323] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:29.727479 32525 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:29.731766 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet aa4b903a93464946ae238ce634b483d0 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:29.732226 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet aa4b903a93464946ae238ce634b483d0 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-aa4b903a93464946ae238ce634b483d0
I20250902 01:58:29.737704 31000 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:29.739686 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:29.741477 32520 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:29.742343 32520 tablet_bootstrap.cc:492] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:29.743146 32520 ts_tablet_manager.cc:1397] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.012s
I20250902 01:58:29.744231 32522 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.746685 32520 raft_consensus.cc:357] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.748248 32520 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:29.749042 32520 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.752262 32520 ts_tablet_manager.cc:1428] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20250902 01:58:29.754456 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-48e0a90ceb7a486aa3a93a73f087dfc6 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:29.754899 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-48e0a90ceb7a486aa3a93a73f087dfc6 on tablet 48e0a90ceb7a486aa3a93a73f087dfc6 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:29.768994 31070 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:29.770155 32219 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:29.774048 32520 ts_tablet_manager.cc:927] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:29.777443 32525 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:29.778563 32525 tablet_bootstrap.cc:492] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:29.779894 32525 ts_tablet_manager.cc:1397] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.053s user 0.020s sys 0.017s
I20250902 01:58:29.782678 32525 raft_consensus.cc:357] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.783449 32525 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:29.784135 32525 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.804193 32520 tablet_copy_client.cc:323] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:40531
I20250902 01:58:29.783636 31011 tablet_copy_source_session.cc:215] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:29.809687 32527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa4b903a93464946ae238ce634b483d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:29.815379 31084 tablet_copy_service.cc:140] P 088424002a894fd5bcc11a06af111445: Received BeginTabletCopySession request for tablet 6c9675a8370e4d4a9f95b3c568abe972 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:45756)
I20250902 01:58:29.815970 31084 tablet_copy_service.cc:161] P 088424002a894fd5bcc11a06af111445: Beginning new tablet copy session on tablet 6c9675a8370e4d4a9f95b3c568abe972 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:45756: session id = 01f6731ce5044a6a948a5ae04da80e6f-6c9675a8370e4d4a9f95b3c568abe972
I20250902 01:58:29.809696 32525 ts_tablet_manager.cc:1428] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250902 01:58:29.821789 32527 tablet_copy_client.cc:806] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:29.823402 31084 tablet_copy_source_session.cc:215] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:29.825418 32527 tablet_copy_client.cc:670] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:29.825749 32520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c9675a8370e4d4a9f95b3c568abe972. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:29.828569 32527 tablet_copy_client.cc:538] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:29.830626 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-805e9f7636dc47059b5af8c090e5ac0a received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:29.831035 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-805e9f7636dc47059b5af8c090e5ac0a on tablet 805e9f7636dc47059b5af8c090e5ac0a with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:29.838249 32527 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:29.843737 32520 tablet_copy_client.cc:806] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:29.844378 32520 tablet_copy_client.cc:670] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:29.848675 32520 tablet_copy_client.cc:538] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:29.867368 32520 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:29.880020 32527 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:29.880942 32527 tablet_bootstrap.cc:492] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:29.881660 32527 ts_tablet_manager.cc:1397] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.019s
I20250902 01:58:29.884088 32527 raft_consensus.cc:357] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.885212 32527 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:29.885854 32527 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.886546 31070 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:58:29.887692 31070 raft_consensus.cc:604] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:29.888772 32527 ts_tablet_manager.cc:1428] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250902 01:58:29.891126 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250902 01:58:29.892367 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:29.892972 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:29.892278 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-aa4b903a93464946ae238ce634b483d0 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:58:29.903096 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-aa4b903a93464946ae238ce634b483d0 on tablet aa4b903a93464946ae238ce634b483d0 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:29.912406 32520 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:29.913017 32520 tablet_bootstrap.cc:492] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:29.913436 32520 ts_tablet_manager.cc:1397] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.047s user 0.017s sys 0.002s
I20250902 01:58:29.915313 32520 raft_consensus.cc:357] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.917157 32520 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:29.917793 32520 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:29.928558 32520 ts_tablet_manager.cc:1428] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20250902 01:58:29.933669 31084 tablet_copy_service.cc:342] P 088424002a894fd5bcc11a06af111445: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-6c9675a8370e4d4a9f95b3c568abe972 received from {username='slave'} at 127.0.0.1:45756
I20250902 01:58:29.933986 31084 tablet_copy_service.cc:434] P 088424002a894fd5bcc11a06af111445: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-6c9675a8370e4d4a9f95b3c568abe972 on tablet 6c9675a8370e4d4a9f95b3c568abe972 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:30.052630 31726 raft_consensus.cc:1215] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20250902 01:58:30.167351 32490 raft_consensus.cc:991] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:30.167856 32427 raft_consensus.cc:1079] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:30.169144 30920 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38810
I20250902 01:58:30.169561 30920 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:30.169834 30920 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:30.177939 30920 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:30.179127 30920 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:30.179754 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:30.181200 30998 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:30.179980 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:30.183995 31070 raft_consensus.cc:3053] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:30.184549 31070 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:30.185202 31070 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:30.185981 31070 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:30.188676 30998 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:30.191900 30889 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:30.192502 32207 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:30.192853 32207 raft_consensus.cc:695] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:30.193429 32207 consensus_queue.cc:237] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:30.194463 31070 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:30.206490 30819 catalog_manager.cc:5582] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:30.227401 31726 raft_consensus.cc:1215] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250902 01:58:30.292670 31726 raft_consensus.cc:1215] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250902 01:58:30.316656 31726 raft_consensus.cc:1215] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250902 01:58:30.500272 32496 raft_consensus.cc:1062] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:30.502193 32496 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.507895 31726 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:30.508503 31072 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:30.509961 32503 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:30.510625 30926 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:30.511663 32503 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:30.512403 32213 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:30.524196 32503 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.525931 30926 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.530022 31070 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.539996 31728 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.541271 30823 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 5 to 6, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.557992 31001 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.563396 31728 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:30.564338 30926 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:30.564446 32213 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:30.565325 32496 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:30.574862 32503 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.575690 30926 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.582484 31728 raft_consensus.cc:2953] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.593560 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 805e9f7636dc47059b5af8c090e5ac0a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:30.595901 30819 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 6 to 7, VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195) evicted. New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.606948 31059 tablet_service.cc:1515] Processing DeleteTablet for tablet 805e9f7636dc47059b5af8c090e5ac0a with delete_type TABLET_DATA_TOMBSTONED (TS 088424002a894fd5bcc11a06af111445 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46532
I20250902 01:58:30.608662 32546 tablet_replica.cc:331] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: stopping tablet replica
I20250902 01:58:30.609371 32546 raft_consensus.cc:2241] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:30.618912 32546 raft_consensus.cc:2270] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:30.622464 32546 ts_tablet_manager.cc:1905] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:30.635200 32546 ts_tablet_manager.cc:1918] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20250902 01:58:30.635488 32546 log.cc:1199] T 805e9f7636dc47059b5af8c090e5ac0a P 088424002a894fd5bcc11a06af111445: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-2-root/wals/805e9f7636dc47059b5af8c090e5ac0a
I20250902 01:58:30.636924 30811 catalog_manager.cc:4928] TS 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): tablet 805e9f7636dc47059b5af8c090e5ac0a (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:30.645553 31001 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:30.646308 32207 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:30.660135 31070 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:30.661414 32207 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:30.686383 32490 raft_consensus.cc:1062] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:30.688551 32490 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.694701 31726 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.695989 32490 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.697314 30926 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.700381 31001 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.700745 32427 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.703256 32490 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.712534 32540 raft_consensus.cc:1062] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:30.716830 32540 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.726629 32427 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.730664 31728 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.731809 30920 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.732113 31001 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250902 01:58:30.730816 31000 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.734244 32489 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.735781 32489 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.738286 31728 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.742436 32540 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:30.749315 30920 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.755172 30819 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.779994 32490 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.783123 32496 raft_consensus.cc:1062] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:30.786840 32496 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.790132 31730 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.790246 30920 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.794261 31001 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.801780 31068 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.808637 30920 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:30.809636 31726 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:30.810021 32489 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:30.811146 32540 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:30.812244 30824 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 3 to 4, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.823086 30924 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:30.824070 31730 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 LEARNER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:30.825603 31068 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250902 01:58:30.825819 32503 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:30.826617 32490 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.831223 32503 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:30.832895 31070 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:30.838338 30998 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:30.838476 31730 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:30.841087 32489 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:30.842726 32503 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:30.851715 32540 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:30.852414 31726 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.855579 30920 raft_consensus.cc:2953] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.863893 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:30.865803 30819 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 4 to 5, VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194) evicted. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:30.869561 32489 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.881295 31001 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.884866 30990 tablet_service.cc:1515] Processing DeleteTablet for tablet 48e0a90ceb7a486aa3a93a73f087dfc6 with delete_type TABLET_DATA_TOMBSTONED (TS 975ecd397f5146409baac87e56a302e9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:30.900954 31729 raft_consensus.cc:2953] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.914971 30811 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6c9675a8370e4d4a9f95b3c568abe972 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:30.940956 30819 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 reported cstate change: config changed from index 4 to 5, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.956218 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:30.975471 32503 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.982841 32565 tablet_replica.cc:331] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: stopping tablet replica
I20250902 01:58:30.984153 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c9675a8370e4d4a9f95b3c568abe972 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:30.983825 30924 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:30.983837 32565 raft_consensus.cc:2241] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:30.992446 32565 raft_consensus.cc:2270] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:30.999485 31070 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.000555 31728 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.008783 30823 catalog_manager.cc:5582] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index 2 to 3, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.017043 32565 ts_tablet_manager.cc:1905] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:31.023437 32567 tablet_replica.cc:331] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:31.024528 32567 raft_consensus.cc:2241] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:31.028973 32567 raft_consensus.cc:2270] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:31.032394 31000 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:31.032871 32565 ts_tablet_manager.cc:1918] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250902 01:58:31.033319 32565 log.cc:1199] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 975ecd397f5146409baac87e56a302e9: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-1-root/wals/48e0a90ceb7a486aa3a93a73f087dfc6
I20250902 01:58:31.035095 30808 catalog_manager.cc:4928] TS 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): tablet 48e0a90ceb7a486aa3a93a73f087dfc6 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:31.035282 32567 ts_tablet_manager.cc:1905] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:31.043151 31070 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:31.045214 32496 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:31.046981 31729 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250902 01:58:31.049853 32496 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:31.053339 32496 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.058029 31070 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.059036 31726 raft_consensus.cc:2953] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.076156 30822 catalog_manager.cc:5582] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index 3 to 4, VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:31.077482 30808 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet aa4b903a93464946ae238ce634b483d0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250902 01:58:31.087513 30924 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:31.088059 30924 raft_consensus.cc:604] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:31.089406 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250902 01:58:31.090802 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:31.091409 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:31.093355 32567 ts_tablet_manager.cc:1918] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250902 01:58:31.093812 32567 log.cc:1199] T 6c9675a8370e4d4a9f95b3c568abe972 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/6c9675a8370e4d4a9f95b3c568abe972
I20250902 01:58:31.095127 30915 tablet_service.cc:1515] Processing DeleteTablet for tablet aa4b903a93464946ae238ce634b483d0 with delete_type TABLET_DATA_TOMBSTONED (TS 4372b4c4c866431faa1bf3e91fd47a88 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38794
I20250902 01:58:31.095271 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet 6c9675a8370e4d4a9f95b3c568abe972 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:31.096098 32567 tablet_replica.cc:331] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: stopping tablet replica
I20250902 01:58:31.097103 32567 raft_consensus.cc:2241] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250902 01:58:31.097782 32567 raft_consensus.cc:2270] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250902 01:58:31.102232 32567 ts_tablet_manager.cc:1905] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250902 01:58:31.154596 32567 ts_tablet_manager.cc:1918] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250902 01:58:31.155159 32567 log.cc:1199] T aa4b903a93464946ae238ce634b483d0 P 4372b4c4c866431faa1bf3e91fd47a88: Deleting WAL directory at /tmp/dist-test-taskNYmR_m/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756778204804806-29783-0/minicluster-data/ts-0-root/wals/aa4b903a93464946ae238ce634b483d0
I20250902 01:58:32.092317 32571 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.092903 32571 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.095057 32571 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.143368 32570 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.160770 32568 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.365694 32570 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.365792 32568 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.368372 32570 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.368736 32568 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.368976 32570 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.369400 32568 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.369655 32570 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.369994 32568 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.250957 32569 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.371809 32569 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.373945 32569 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.416384 32568 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.371381 32570 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.522704 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1407) took 1396 ms. Trace:
I20250902 01:58:32.523247 31070 rpcz_store.cc:276] 0902 01:58:31.126798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:32.521904 (+1395106us) service_pool.cc:225] Handling call
0902 01:58:32.522207 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:32.522381 (+ 174us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:32.522403 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:32.522439 (+ 36us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:32.522467 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:32.522514 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:32.522650 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:32.538841 32574 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.539454 32574 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.541836 32574 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.551962 32572 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.552234 32577 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.552488 32572 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.552771 32577 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.553877 32572 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.554388 32577 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.555657 32568 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.555823 32578 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.556174 32568 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.556440 32578 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.558490 32578 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.558744 32568 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.559000 32578 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.559477 32568 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.559645 32578 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.560034 32568 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.561913 32578 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.562366 32568 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.559681 32577 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.563079 32577 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.563756 32570 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.564415 32570 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.565510 32577 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.566589 32570 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.578205 32581 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.578475 32570 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.581307 32580 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.578965 32581 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.586126 32577 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.586330 32579 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.586737 32577 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.586866 32579 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.587477 32581 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.588125 32580 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.588814 32577 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.589515 32577 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.584837 32569 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.589927 32577 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.579224 32583 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.590652 32569 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.591847 32583 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.593055 32569 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.589987 32580 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.579062 32570 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.597239 32583 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.593588 32575 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.597647 32570 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.605218 32583 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.605803 32583 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.608233 32583 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.597913 32575 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.612035 32575 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.612552 32575 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.613013 32575 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.617924 32577 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.616406 32583 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.618678 32570 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.625437 32576 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.622049 32574 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.625978 32576 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.626451 32574 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.629292 32574 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.629565 32577 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.623889 32579 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.623366 32588 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.630074 32577 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.633312 32577 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.634176 32585 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.626741 32570 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.641654 32570 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.629346 32576 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.642280 32583 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.624439 32575 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.634639 32585 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.715880 32585 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.630471 32588 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.721879 32588 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.705389 32574 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.723524 32574 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.725114 32574 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.728174 32583 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.735066 32592 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.735602 32592 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.737462 32587 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.738094 32592 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.738761 32592 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.738166 32587 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.740275 32591 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.745638 32575 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.746197 32575 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.748586 32575 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.749145 32582 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.749620 32582 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.762847 32587 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.740770 32591 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.739265 32592 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.785609 32586 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.761436 32576 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.773123 32574 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.789929 32573 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.790095 32572 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.790393 32573 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.790603 32572 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.792312 32573 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.792574 32572 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.795198 32578 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.795631 32578 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.786069 32586 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.801532 32568 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.801585 32577 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.802058 32568 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.802114 32577 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.802373 32592 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.798060 32587 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.793915 32576 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.804756 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 670) took 1664 ms. Trace:
I20250902 01:58:32.809293 31726 rpcz_store.cc:276] 0902 01:58:31.140666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:31.160241 (+ 19575us) service_pool.cc:225] Handling call
0902 01:58:31.160535 (+ 294us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:31.160705 (+ 170us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:31.160728 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:31.160763 (+ 35us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:31.160797 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:31.160854 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:32.804683 (+1643829us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:32.793313 32574 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.799939 32572 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.800480 32579 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.800989 32575 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.803905 32589 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.805425 32585 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.804118 32593 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.801600 32598 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.810765 32587 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.814343 32579 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.886153 32591 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.886471 32579 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.886713 32591 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.814656 32572 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.887248 32591 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.888953 32572 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.833555 32568 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.889251 32591 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.889930 32574 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.886575 32587 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.812865 32592 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.875712 32582 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.834151 32590 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.834061 32595 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.860957 32586 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.887144 32579 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.816717 32585 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.897792 32601 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.875365 32576 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.890148 32591 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.876183 32578 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.874500 32600 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.859194 32577 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.833942 32594 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.893754 32574 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.815785 32575 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.902323 32580 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.829586 32596 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.912598 32591 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.829459 32593 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.913287 32581 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.914134 32573 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.901296 32592 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.876932 32598 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.816051 32589 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.909847 32579 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.922390 32580 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.916116 32574 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.916352 32600 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.927431 32597 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.911422 32590 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.923472 32577 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.930197 32573 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.913549 32595 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.939872 32604 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.940896 32605 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.942327 32611 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.943957 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.943604 32612 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.944664 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.966240 30924 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:32.944763 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.940948 32606 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.937181 32570 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:32.949855 32586 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.948212 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.954480 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.954138 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.970945 30922 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:32.936535 32581 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.954560 32598 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.980120 32613 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:32.964085 32572 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.916244 32601 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.984190 32610 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:32.985357 32593 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:32.963730 32597 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.953997 32577 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.953262 32599 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.997838 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.925056 32596 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.972697 32570 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.918848 32594 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.997401 32580 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:32.977648 32586 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.977377 32604 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:32.993065 32572 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.002135 32576 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.038641 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1408) took 1898 ms. Trace:
I20250902 01:58:32.990756 32602 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.000895 32598 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.041826 31068 rpcz_store.cc:276] 0902 01:58:31.138428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:32.900432 (+1762004us) service_pool.cc:225] Handling call
0902 01:58:32.900698 (+ 266us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:33.002443 (+101745us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:33.002471 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:33.002508 (+ 37us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:33.002532 (+ 24us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:33.036416 (+ 33884us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400040e54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:33.036456 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:33.036551 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233680640}
I20250902 01:58:33.033757 32585 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.033383 32593 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.032058 32601 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.019076 32590 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.046554 32570 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.049052 32594 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.032523 32618 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.001669 32610 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.051605 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.044425 32600 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.019874 32614 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:32.979444 32606 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.037789 32575 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.044668 32591 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:32.979086 32611 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.056097 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:32.979468 32605 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.020112 32599 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.000241 32613 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:32.992635 32612 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:33.283044 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 331404us
I20250902 01:58:33.362465 32621 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.363451 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.365092 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.372848 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.050892 32576 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.052153 32598 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.258936 32581 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.054944 32602 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.057260 32593 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.105847 32579 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:33.426285 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 53857.8us
I20250902 01:58:33.544662 32599 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.409711 32618 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.550402 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.550966 31723 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:33.551123 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.173533 32596 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.551237 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.315107 32580 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.551784 31721 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:33.511003 32624 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:33.552065 32580 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.552621 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.553292 31724 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:33.553639 32618 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.203229 32586 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.554952 32605 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.552480 32624 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.492830 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.499192 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.426616 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.493687 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.079834 32592 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.363180 32574 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.256502 32604 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.277863 32570 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.351578 32616 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:33.272992 32587 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.553947 32580 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.116289 32568 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:33.256081 32585 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.353870 32590 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.240507 32608 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:33.172329 32577 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.284756 32583 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.152134 32607 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:33.555092 31041 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.008s
I20250902 01:58:33.152580 32595 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:33.638391 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.008s sys 0.008s
W20250902 01:58:33.652344 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.008s sys 0.011s
W20250902 01:58:33.652602 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 660822us
I20250902 01:58:33.654335 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.654611 31693 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:33.654820 32598 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.564570 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.557070 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.579375 31730 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:33.592823 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.607921 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.656330 31000 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
W20250902 01:58:33.639719 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.021s sys 0.013s
I20250902 01:58:33.175805 32582 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:33.239089 32573 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.657315 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.202128 32609 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.660351 31693 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:33.315629 32597 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:33.585431 31751 debug-util.cc:398] Leaking SignalData structure 0x7b080028bde0 after lost signal to thread 29786
I20250902 01:58:33.585651 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.587148 32568 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:33.663268 31041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.000s sys 0.010s
I20250902 01:58:33.585820 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.581959 32587 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.664408 31727 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:33.664642 32568 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.589781 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.585675 32610 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.588168 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.588522 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.586024 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.115725 32603 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.670156 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.670629 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.670934 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.671461 30996 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.671729 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.372788 32600 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:33.670181 31721 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.672437 32600 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.672801 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.673971 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.674290 32600 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.675813 32577 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.657596 32582 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.657665 32583 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.670087 32587 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.678195 32582 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.678292 32583 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.678884 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.680125 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.680317 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.456406 32572 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.060711 32589 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.683223 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.657192 32617 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:33.661772 32595 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:33.685899 32595 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.686293 32595 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:33.687532 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.661974 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.554919 32602 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.582176 32576 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.692939 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.588222 32574 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:33.659605 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.581049 32570 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.662585 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.698019 32617 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.706728 32617 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:33.663221 30992 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.509778 32619 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.707091 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.663708 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:33.663887 31751 debug-util.cc:398] Leaking SignalData structure 0x7b0800233760 after lost signal to thread 30858
I20250902 01:58:33.664227 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.456679 32611 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.667181 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.665282 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.697360 32609 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.724180 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.667901 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.657122 32607 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.668818 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.669301 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.736169 31729 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:33.634371 32616 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.736934 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.676344 32577 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.681205 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.638530 32613 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.527016 32625 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.739992 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.670542 32603 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.425855 32621 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.588099 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.691753 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.691854 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.596451 32612 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.693607 30924 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.672313 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.615440 32585 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.681878 32568 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:33.634215 32590 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.707741 30923 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:33.724372 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.708236 32570 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.705013 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.456610 32606 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.738420 30920 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
W20250902 01:58:33.658164 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s user 0.021s sys 0.018s
I20250902 01:58:33.707809 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.707125 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.705963 32574 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.707448 32619 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.413383 32614 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.657258 32608 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.916739 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.944029 32583 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.944140 30998 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.918244 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.944548 31066 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.944546 32583 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:33.945624 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.945693 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.946446 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.946746 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.946834 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.947249 32573 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.916301 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.946545 32583 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.740376 32625 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.947688 32573 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.947388 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.948490 31001 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:33.884298 32595 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.949509 32573 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.916989 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.959242 32608 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.949757 32625 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.949919 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.959892 32595 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.960188 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.960453 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.960577 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.961051 31070 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.961091 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.961416 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.961766 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.962004 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.962273 31070 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.962340 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.962759 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.963094 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.963263 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.950230 30809 catalog_manager.cc:4928] TS 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): tablet aa4b903a93464946ae238ce634b483d0 (table test-workload [id=7afa88e279034b01a4a86f528bb93af2]) successfully deleted
I20250902 01:58:33.964537 32577 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.847860 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.883291 32617 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.863804 32624 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.862977 32616 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.975450 32617 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.975549 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.952304 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.975873 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.976842 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.740334 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.953387 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.954243 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.954306 32614 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.872490 32593 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.955179 30998 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:33.910360 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.956146 32619 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.810329 32607 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.837426 32609 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:33.902391 31751 debug-util.cc:398] Leaking SignalData structure 0x7b080034c1a0 after lost signal to thread 30950
W20250902 01:58:33.983283 31751 debug-util.cc:398] Leaking SignalData structure 0x7b0800173a40 after lost signal to thread 31025
I20250902 01:58:33.983569 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:33.984233 31751 debug-util.cc:398] Leaking SignalData structure 0x7b08002f78c0 after lost signal to thread 31099
I20250902 01:58:33.984423 32573 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:33.984751 32573 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:33.985054 32573 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
W20250902 01:58:33.985294 31751 debug-util.cc:398] Leaking SignalData structure 0x7b0800130a40 after lost signal to thread 31754
I20250902 01:58:33.985167 32621 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.986701 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.957839 32590 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:33.788128 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:33.964285 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.777241 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.993810 30964 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:33.994342 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.994716 30964 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:33.994843 31072 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.995462 30964 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:33.995620 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.996610 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.997442 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.965004 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.963505 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.740965 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.896883 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.907191 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.000222 31073 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:34.001256 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.950071 32568 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:33.967820 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.002897 32609 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.004844 32609 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.005154 32609 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.006415 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.006731 30964 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:34.008286 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.009548 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.009755 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.969287 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.969457 32574 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:33.740407 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.969877 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.012880 32625 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:33.897595 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.014242 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.014339 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.014537 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.015288 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.015985 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.016387 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.017108 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.018123 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.862777 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.019881 31069 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:33.976924 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.020843 31067 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.953022 32585 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.974997 30917 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:34.021723 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.021875 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.021785 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.022030 32609 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.022709 31067 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:34.022939 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.981338 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.023298 31068 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:34.023828 31725 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:34.023941 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.023847 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.024677 31067 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:34.024719 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.024971 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.025518 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.025765 31730 raft_consensus.cc:1238] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:34.026032 31067 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:34.026664 31730 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:33.987972 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.986629 32603 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:33.988951 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.989224 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.033941 31693 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:33.991856 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.992125 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.992516 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:33.992820 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.037987 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.038672 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.038937 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.039744 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.992878 32573 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.002429 30892 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.002373 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.003206 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.003952 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.998836 32619 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.009764 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:33.968184 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.010429 30920 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:34.044629 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.011286 32624 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.012776 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.013250 32625 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:33.970803 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.978681 30921 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:34.019308 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:33.768258 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.028564 30922 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:34.029050 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.049432 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.049192 30923 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:34.049513 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.022755 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.022086 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.050598 31070 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:33.774338 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.048669 32625 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.033130 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.989617 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.035171 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.035701 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:34.036857 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.053412 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.039834 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:33.947398 31066 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:34.041221 31693 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.055950 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:34.056070 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.056514 31693 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.034098 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.041735 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.047502 32583 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.040768 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.042378 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.040519 30925 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:34.045192 30917 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:34.043579 32602 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.049202 32609 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.052574 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.054282 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.059062 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.059409 31692 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.058838 32612 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.060248 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.047353 32619 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.048736 32624 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.070982 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.071080 32619 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.071177 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.060482 32595 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.072244 31692 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.059950 32583 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.062536 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.062860 31693 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.075109 30922 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.062592 32602 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.063058 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.077349 30925 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:34.064766 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.060149 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.079192 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.071007 32624 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.081082 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.061317 32613 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:34.060562 32585 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.069985 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.074466 32617 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.062373 32593 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:34.074204 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:34.057812 30892 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.029404 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.078265 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.072321 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.061627 32579 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.081096 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.070528 32612 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.068404 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.074275 32587 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.073460 32595 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.075492 32583 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.079023 32621 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.072888 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.078930 32602 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.086468 31692 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.069196 32616 consensus_queue.cc:1046] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20250902 01:58:34.088294 30892 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.090386 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.096980 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.088840 32582 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.087538 32579 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.093214 32595 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.099031 32572 raft_consensus.cc:3053] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:34.094620 32621 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.083119 32585 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.105782 32621 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.106799 30922 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:34.076833 32608 consensus_queue.cc:1046] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250902 01:58:34.107484 30920 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:34.107924 32608 raft_consensus.cc:3053] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:34.108291 32608 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:34.108882 32625 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.109058 32608 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.110167 32608 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.105907 32585 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.099293 32579 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.111354 31693 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.103806 32573 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.091867 30917 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:34.090101 32612 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.091763 32570 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.113860 32570 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:58:34.089111 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 48072.5us
I20250902 01:58:34.094746 32568 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.098492 32582 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.099782 30921 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:34.098255 32587 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.098457 32627 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.100824 30918 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:34.104110 32572 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:34.082500 32613 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.093889 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.097265 30892 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.113206 30925 raft_consensus.cc:1238] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:34.119057 30921 raft_consensus.cc:604] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:34.119617 32627 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.121011 32627 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.116036 32570 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.114019 31693 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.099104 31692 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.114780 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.113334 32617 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:34.116602 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.116873 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.123979 31692 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.121706 32613 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:34.117709 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.118003 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.121639 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250902 01:58:34.118880 32582 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.117165 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:34.125339 32592 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.122822 32597 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:34.133435 32592 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.123718 32587 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:34.126668 32617 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:34.123921 32573 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.122262 32625 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.136538 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:34.140197 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:34.122704 32621 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.135743 32572 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.141301 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.141345 32583 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.141883 32587 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:34.143496 31692 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.150226 30892 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:34.137904 32592 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.151453 32617 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.138422 32573 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.151646 32587 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.154278 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.159480 32636 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.156227 30889 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.159981 32636 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.160476 32636 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.165081 30924 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.165773 31692 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:34.188158 32612 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.190524 32612 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.155238 32573 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.164938 32572 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:34.154930 32635 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.196388 32635 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.194962 32643 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.195641 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.197535 32635 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.201920 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.205214 31692 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.207314 31692 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.208702 32632 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.199267 32643 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.207681 32644 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.211998 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.213079 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.211398 32643 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.211519 32621 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.211983 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.210872 31692 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.240159 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.243952 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.244082 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.217650 32637 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.244699 32637 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.245025 32637 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.245131 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.251225 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.256903 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:34.261462 31692 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.264178 32617 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:58:34.264828 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 108713us
I20250902 01:58:34.266863 32647 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.267925 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:34.256022 32621 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:34.280650 30823 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 2 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:34.282637 32602 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.285293 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.286891 32602 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.267130 32592 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.301499 32582 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.303301 32582 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.303819 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.304879 32637 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.306484 32637 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.272526 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.297941 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.318727 31695 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:58:34.319033 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:34.320716 31695 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.322422 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.324769 31695 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.332003 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.325912 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.335160 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.335969 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.337107 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.338253 32619 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.339772 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.340162 32644 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.340904 32621 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.340240 32643 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.340607 32613 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.341261 32621 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.341684 32621 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.342635 32613 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:34.343176 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.340624 32644 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.344209 32644 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.330303 32573 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.345876 32573 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.347625 31695 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.352272 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.352546 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.356263 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.327610 32582 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.325793 32583 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.330065 32627 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.325714 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.312283 32579 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.364131 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:34.364697 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:34.328697 32592 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:34.346570 32643 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.346303 32636 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.365759 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:34.517398 31039 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.538s (SENT) [suppressed 1 similar messages]
W20250902 01:58:34.520663 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 187626us
I20250902 01:58:34.521239 30967 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:34.499754 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 112333us
I20250902 01:58:34.522409 31692 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.362351 32587 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.325436 32642 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.340668 32617 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.503731 32582 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.505743 32619 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.507292 32621 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.507828 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:34.509938 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 211561us
I20250902 01:58:34.513849 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.514166 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.514782 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.511229 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.515326 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.525069 32593 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.526324 32643 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.519748 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:34.528084 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 377945us
I20250902 01:58:34.331429 32585 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.531981 31073 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.501068 32602 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.536660 30967 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:34.568326 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 54554.7us
I20250902 01:58:34.528045 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.531997 32609 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:34.578989 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.532671 32587 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.585969 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.532969 32617 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.534251 32582 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.566614 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:34.523190 32595 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.573318 32616 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.625491 32621 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.625991 32595 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.583001 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:34.616081 32579 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.616889 32570 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:34.585947 32593 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.596836 32617 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.583097 32602 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:34.634109 32570 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.587339 32587 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.597224 32568 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.644644 32635 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.645437 32617 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.646492 32635 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:34.638650 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 59456.5us
I20250902 01:58:34.647217 32617 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.623018 32572 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.519153 32624 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.637109 32602 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.648195 32585 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.589422 30892 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.648154 30967 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.588272 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:34.666738 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 78533.1us
I20250902 01:58:34.646924 31067 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:34.635476 32582 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:34.660074 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 70742.7us
W20250902 01:58:34.672616 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 155261us
I20250902 01:58:34.667976 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.665391 30967 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:34.672927 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 97055.1us
I20250902 01:58:34.671074 31070 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:34.714298 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:34.718922 30889 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.720506 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.720813 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.721820 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.663653 32646 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250902 01:58:34.728688 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 187752us
I20250902 01:58:34.723268 30889 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.726416 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.730680 30967 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.732122 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.669540 32574 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.671470 32572 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.733129 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.735301 30967 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:34.732854 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 64961.9us
I20250902 01:58:34.736824 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.737768 31075 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:34.737888 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.740128 32579 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.734546 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.727653 32653 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.743530 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:34.743136 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 225347us
I20250902 01:58:34.734862 32568 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.750051 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.747633 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.739251 32585 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:58:34.747903 30889 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.754797 30918 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:34.743065 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.755684 30967 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.742403 32657 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.760012 32624 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:34.760600 32662 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.760797 32636 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.762235 32627 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.762262 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.763402 30889 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.764400 32651 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.746245 32572 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.728143 32646 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:34.733851 32609 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.755841 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.753772 32637 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.748454 32574 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.771250 32585 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.765378 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.769011 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.757323 32654 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:34.773290 32637 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:34.775637 30967 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.776654 31038 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.778160 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.771679 32659 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.779453 32574 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.779847 30918 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:34.775007 32617 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20250902 01:58:34.779784 32572 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.781062 32621 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:34.771502 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:34.770555 32646 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.789896 31038 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.789451 30889 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.792239 30889 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.812553 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:34.838534 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 46368.1us
W20250902 01:58:34.838928 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:34.794077 32572 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.784451 32579 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:34.811708 32651 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.831434 31728 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:34.832737 31721 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:34.843894 31729 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:34.845149 31727 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.845665 31695 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.848042 32602 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.849133 32582 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.864164 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.864305 32585 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.855476 31724 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.865334 32602 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.853672 31722 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.853782 31725 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.854710 31730 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.854822 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:34.867458 32593 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:34.855998 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.856537 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.857234 32637 raft_consensus.cc:2762] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250902 01:58:34.852697 30967 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.783079 32644 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.850524 32587 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.863766 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.869369 31073 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:34.872294 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.873989 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.874661 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.874794 31069 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:34.868448 32593 consensus_queue.cc:1046] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20250902 01:58:34.882221 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.882409 32657 raft_consensus.cc:3053] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:34.888993 32657 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:34.889925 32657 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.891325 32657 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.862385 31695 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.886873 32614 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.892076 31726 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.892314 31695 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.893281 31695 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.893811 30967 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.894258 31695 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.894366 32582 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.896102 31000 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:34.893334 32602 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.882995 32648 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.884563 30889 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.900013 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.901194 30889 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.905711 30889 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.912000 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.913014 32654 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.915688 32607 consensus_queue.cc:1046] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:34.916682 32607 raft_consensus.cc:3053] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:34.917539 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.917976 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.918327 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.916988 32607 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:34.914252 32577 consensus_queue.cc:1046] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20250902 01:58:34.915784 32608 consensus_queue.cc:1046] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:34.918236 31072 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:34.915012 30889 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.907172 32583 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:34.909391 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:34.909098 32614 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.908018 32617 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.930217 31695 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.919299 31723 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:34.931437 31695 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:34.907238 32585 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.914100 32640 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:34.942322 32640 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.942788 32640 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:34.914108 32627 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:34.936743 30889 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.937466 31695 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:34.938071 32595 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.907933 32572 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250902 01:58:34.939832 32573 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.908718 32590 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20250902 01:58:34.939956 32655 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20250902 01:58:34.948331 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:34.913357 32579 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250902 01:58:34.914135 32652 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:34.946961 32616 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:34.951400 32644 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:34.936717 32621 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.963446 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.928905 32583 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:34.919796 32568 consensus_queue.cc:1046] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250902 01:58:34.961280 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.974185 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.949524 32585 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.949355 32570 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:34.984097 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:34.927038 32607 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:34.978412 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.976469 32583 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:34.975133 32627 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:34.932754 32642 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20250902 01:58:34.976902 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:34.990952 32603 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.000536 32654 raft_consensus.cc:991] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:35.004179 30926 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:35.005502 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.990924 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:34.995457 32607 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.010560 30917 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.006757 32587 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:35.010823 32651 raft_consensus.cc:1079] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:58:35.002681 32603 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.010411 32619 consensus_queue.cc:1046] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20250902 01:58:35.024389 32640 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.036533 32646 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.049921 30917 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.051107 32583 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.031177 32659 consensus_queue.cc:1046] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20250902 01:58:35.052697 32624 raft_consensus.cc:3053] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.052999 32624 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.053093 32583 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:35.053802 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.054674 32659 raft_consensus.cc:3053] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.055029 32659 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.053678 32624 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.056164 32659 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.056875 32624 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.057415 32659 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.066521 30923 raft_consensus.cc:1238] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.070215 32612 raft_consensus.cc:3053] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.070554 32612 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.071247 32612 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.072338 32612 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.076054 32608 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.076731 32616 raft_consensus.cc:3053] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.077118 32616 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.076751 32608 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.077975 32616 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.079159 32616 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.078780 32608 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.080576 32608 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.085526 32646 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:35.086607 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.086241 32603 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.088330 32603 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:35.093322 32669 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.074s user 0.003s sys 0.000s
I20250902 01:58:35.098194 32651 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:35.105059 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.101470 31075 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:35.132937 30822 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 reported cstate change: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:35.136662 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.151213 32568 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.155345 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.156354 32568 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.160799 32572 raft_consensus.cc:3053] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.160331 32568 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.161139 32572 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:35.169684 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:35.173398 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:35.174110 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 256236us
I20250902 01:58:35.174836 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.176183 32572 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.177524 32568 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.177611 32572 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.179070 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.180291 31695 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.181514 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.182569 32587 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.181387 32621 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.179697 32640 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:35.202817 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.183802 32642 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.188819 30965 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.202052 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.202225 31039 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.204583 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.206039 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.206902 30965 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.206975 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.207463 30997 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250902 01:58:35.208171 30965 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.208307 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.208626 31039 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.209511 31039 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.210013 32590 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.211706 32614 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.212031 31726 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.213907 32670 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.214550 32577 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.216169 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.217105 32619 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.217242 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.217905 31727 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51240
I20250902 01:58:35.218112 31722 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:35.218451 31727 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:35.218827 31727 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.218878 32579 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.221685 30996 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:35.221975 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.223176 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.223783 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:35.224339 31730 raft_consensus.cc:2391] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:35.224287 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.225082 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.225200 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.228606 32577 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.229864 32577 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.231079 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.232645 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.233592 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.234694 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.235574 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.236027 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.232402 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.232395 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.237232 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.239389 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.239686 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.240487 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.240605 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.241586 30993 raft_consensus.cc:1238] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:35.242416 30924 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:35.239455 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.243719 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.249087 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:35.250646 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:35.252635 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 44536.3us
I20250902 01:58:35.251121 31695 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.255728 30889 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.258960 30965 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:35.247839 32587 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.260772 30965 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.262051 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.264184 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.264827 32583 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.265089 32637 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.266275 32673 consensus_queue.cc:1046] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250902 01:58:35.268767 30965 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.269495 32670 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.269843 31725 raft_consensus.cc:1238] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:35.269490 32673 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.260298 32602 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.267052 32657 consensus_queue.cc:1046] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250902 01:58:35.263464 32624 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:35.253494 32590 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:35.270716 32624 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:35.270982 32657 raft_consensus.cc:3053] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.271109 32624 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.271309 32657 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.272025 32657 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.270349 32673 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.274647 32673 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.274269 30889 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.266939 32619 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.278951 32673 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.274652 32657 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.260205 32640 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250902 01:58:35.278990 32583 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.281901 32651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:57726 (request call id 11) took 1283 ms. Trace:
I20250902 01:58:35.275255 30965 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.285805 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.298790 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.292544 31722 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.301024 32670 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.301833 30995 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.287431 32645 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.302711 32577 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:35.300384 30965 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.300815 32616 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.305265 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.305837 32670 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.306329 32619 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.307615 30965 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.295908 32651 rpcz_store.cc:276] 0902 01:58:33.978255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:34.095034 (+116779us) service_pool.cc:225] Handling call
0902 01:58:34.132507 (+ 37473us) log.cc:844] Serialized 506 byte log entry
0902 01:58:34.527800 (+395293us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b1800009120. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60216d321 00007fc6020c0db1 00007fc6020a9502 00007fc6020ae911 00007fc6020ae899 00007fc6020ae829 00007fc6020ae7f1 00007fc6020ad9bd 0000000000379254 0000000000379099
0902 01:58:34.831070 (+303270us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b4c00291580. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6004c8940 00007fc6004d9772 00007fc6004d9cd2 00007fc60049e78b 00007fc6004cbe5c 00007fc6004cc31e 00007fc600738ceb 00007fc6020ab251 00007fc6020a9859 00007fc6020ae911 00007fc6020ae899
0902 01:58:35.261295 (+430225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12727,"raft.queue_time_us":274747,"raft.run_cpu_time_us":42439,"raft.run_wall_time_us":756879,"spinlock_wait_cycles":69819392,"thread_start_us":144063,"threads_started":3,"wal-append.queue_time_us":48113,"wal-append.run_cpu_time_us":1559,"wal-append.run_wall_time_us":1003080}
I20250902 01:58:35.309680 32619 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.310034 32670 consensus_queue.cc:1046] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:35.310977 32619 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.311318 32619 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.313733 32577 consensus_queue.cc:1046] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250902 01:58:35.315918 32572 raft_consensus.cc:3053] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.316238 32572 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.316088 32619 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.317250 32619 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.317025 32572 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.317673 32624 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.318148 32572 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.321663 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.310981 31727 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.331836 31727 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:35.331733 32624 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:35.332603 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.332643 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:35.333137 30995 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.333433 30923 raft_consensus.cc:3053] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.333807 30923 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.334540 30923 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.336226 30923 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.338354 30995 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:35.339313 31692 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:35.312228 30965 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.346844 30823 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40172 (request call id 238) took 1086 ms. Trace:
I20250902 01:58:35.351667 30889 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.351056 30819 catalog_manager.cc:5582] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 3 to 4, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 4 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:35.349464 32670 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.354281 32583 consensus_queue.cc:1046] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:35.348690 32617 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:35.355602 32617 raft_consensus.cc:695] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:35.355141 32640 raft_consensus.cc:3053] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.356483 32640 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.356388 32617 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.356701 31075 consensus_queue.cc:237] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:35.357230 32640 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.360463 30889 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.362545 32646 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.364970 30923 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:35.367183 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.367496 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:35.363157 32657 consensus_queue.cc:1046] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250902 01:58:35.373953 32619 consensus_queue.cc:1046] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:35.366707 32652 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.374248 32657 raft_consensus.cc:3053] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.374567 32657 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.374776 32612 raft_consensus.cc:3053] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.374868 32646 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.375146 32612 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.375438 32640 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.375206 32670 consensus_queue.cc:1046] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:35.376408 32670 raft_consensus.cc:3053] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.376016 32612 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.376777 32670 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.378108 32612 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.378006 32670 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.379097 32670 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.383625 30965 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.384603 32619 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.385327 30965 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.386124 32619 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.387100 30965 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.375187 32657 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.387817 32619 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.388748 32657 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.389034 30965 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.389670 32619 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.347296 30823 rpcz_store.cc:276] 0902 01:58:34.260368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:34.260487 (+ 119us) service_pool.cc:225] Handling call
0902 01:58:35.346744 (+1086257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:34.365925 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:34.654930 (+289005us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:34.660603 (+ 5673us) write_op.cc:435] Acquired schema lock
0902 01:58:34.664523 (+ 3920us) tablet.cc:660] Decoding operations
0902 01:58:34.740876 (+ 76353us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:34.740946 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:34.740971 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0902 01:58:34.741144 (+ 173us) tablet.cc:699] Row locks acquired
0902 01:58:34.741160 (+ 16us) write_op.cc:260] PREPARE: finished
0902 01:58:34.741277 (+ 117us) write_op.cc:270] Start()
0902 01:58:34.741371 (+ 94us) write_op.cc:276] Timestamp: P: 1756778314741241 usec, L: 0
0902 01:58:34.741390 (+ 19us) op_driver.cc:348] REPLICATION: starting
0902 01:58:34.741948 (+ 558us) log.cc:844] Serialized 536 byte log entry
0902 01:58:35.239639 (+497691us) op_driver.cc:464] REPLICATION: finished
0902 01:58:35.339203 (+ 99564us) write_op.cc:301] APPLY: starting
0902 01:58:35.339277 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:35.339355 (+ 78us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:35.339450 (+ 95us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:35.343979 (+ 4529us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:35.344163 (+ 184us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0902 01:58:35.344227 (+ 64us) write_op.cc:312] APPLY: finished
0902 01:58:35.344455 (+ 228us) log.cc:844] Serialized 23 byte log entry
0902 01:58:35.344659 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:35.344745 (+ 86us) write_op.cc:454] Released schema lock
0902 01:58:35.344790 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:35.345802 (+ 1012us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:35.345836 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":99369,"apply.run_cpu_time_us":2310,"apply.run_wall_time_us":6552,"num_ops":1,"prepare.queue_time_us":33838,"prepare.run_cpu_time_us":44655,"prepare.run_wall_time_us":560780,"raft.queue_time_us":262929,"raft.run_cpu_time_us":20825,"raft.run_wall_time_us":112642,"replication_time_us":498085,"thread_start_us":357818,"threads_started":4,"wal-append.queue_time_us":194395}]]}
I20250902 01:58:35.390673 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.391407 32619 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.391937 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.393355 32652 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.396847 32652 consensus_queue.cc:1046] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:35.397120 32654 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:35.397992 32652 raft_consensus.cc:3053] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.398285 32654 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.398456 32652 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.398694 32654 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.399904 30889 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:35.399521 32654 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.399505 32652 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.410005 32654 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.412339 30997 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250902 01:58:35.415694 31038 consensus_peers.cc:489] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92f16b8a15144fcf8807608bdee71123. This is attempt 1: this message will repeat every 5th retry.
W20250902 01:58:35.417420 30889 consensus_peers.cc:489] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf39ccba06f4a0998fb41dd68a0702a. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:35.414366 32645 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.420459 32568 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250902 01:58:35.421386 32652 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.422048 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:35.438872 32645 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.439442 32640 consensus_queue.cc:1046] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20250902 01:58:35.450039 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.450732 32568 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:35.452454 32637 raft_consensus.cc:3053] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:35.452795 32637 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.453424 32637 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.454330 31727 raft_consensus.cc:1238] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 3. Ops: []
I20250902 01:58:35.454490 32637 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:35.454655 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.455694 31039 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.448000 32583 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250902 01:58:35.456621 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.456915 32583 raft_consensus.cc:3053] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.457230 32583 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.459322 32672 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.459919 32672 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.460453 32672 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.462260 31039 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.462908 32568 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.463882 31039 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.464102 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:35.463956 32583 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.464509 30926 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.464517 32568 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.465102 32583 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.465631 31039 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.466197 32568 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.467111 31039 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.467588 30926 raft_consensus.cc:1238] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.467684 32568 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.468850 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.469473 32568 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:35.469486 30965 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.474017 32657 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.474597 32657 consensus_queue.cc:1046] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:35.475126 30926 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:58:35.475454 32670 raft_consensus.cc:3053] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.475831 30926 raft_consensus.cc:604] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.475917 32670 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.476806 32670 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.480341 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
W20250902 01:58:35.473554 32683 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20250902 01:58:35.481045 32670 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.474864 31039 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.483842 31039 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.484715 32673 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:35.482688 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:35.485245 32568 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.485877 31039 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.487592 32673 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.486639 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:35.488081 31039 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:35.494856 32673 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.496672 32673 consensus_queue.cc:1046] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:35.498188 32672 raft_consensus.cc:3053] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.498528 32672 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.499347 32672 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:35.501891 32672 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.505234 30819 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 3, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:35.511546 30997 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:35.515875 32568 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:35.593010 32637 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:35.593487 32637 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.595923 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:35.596696 30997 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:35.599048 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:35.599635 31066 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:35.600802 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.601568 32583 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:35.601920 32583 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:35.602246 32583 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.609535 32637 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:35.609164 32583 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.611487 32583 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:35.612612 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:35.613338 30997 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:35.613629 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:35.617668 30926 raft_consensus.cc:1238] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:35.620524 31066 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:35.621541 32670 consensus_queue.cc:1046] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:35.621635 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:35.622430 32670 raft_consensus.cc:3053] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.622591 32583 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:35.623174 32583 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.624115 32583 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.626133 32670 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.626909 32670 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.628214 32670 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.640861 30819 catalog_manager.cc:5582] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:35.650764 32688 ts_tablet_manager.cc:927] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:58:35.652182 32688 tablet_copy_client.cc:323] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:58:35.653291 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 5cf39ccba06f4a0998fb41dd68a0702a from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:58:35.653653 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 5cf39ccba06f4a0998fb41dd68a0702a from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-5cf39ccba06f4a0998fb41dd68a0702a
I20250902 01:58:35.657919 30936 tablet_copy_source_session.cc:215] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:35.660900 32688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf39ccba06f4a0998fb41dd68a0702a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:35.672981 32688 tablet_copy_client.cc:806] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:35.673565 32688 tablet_copy_client.cc:670] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:35.676527 32688 tablet_copy_client.cc:538] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:58:35.682583 32688 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:35.700958 32688 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250902 01:58:35.701725 32688 tablet_bootstrap.cc:492] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:35.702250 32688 ts_tablet_manager.cc:1397] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.001s
I20250902 01:58:35.704542 32688 raft_consensus.cc:357] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:35.705230 32688 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:35.705852 32688 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:35.708787 32688 ts_tablet_manager.cc:1428] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20250902 01:58:35.710073 32583 raft_consensus.cc:991] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: : Instructing follower 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:35.710512 32583 raft_consensus.cc:1079] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Signalling peer 975ecd397f5146409baac87e56a302e9 to start an election
I20250902 01:58:35.711058 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-5cf39ccba06f4a0998fb41dd68a0702a received from {username='slave'} at 127.0.0.1:57742
I20250902 01:58:35.711527 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-5cf39ccba06f4a0998fb41dd68a0702a on tablet 5cf39ccba06f4a0998fb41dd68a0702a with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:35.714121 30997 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:53772
I20250902 01:58:35.714608 30997 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:35.715235 30997 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.732199 30997 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.734385 30997 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:35.736048 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:35.736534 30926 raft_consensus.cc:3053] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:35.736806 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:35.737624 30926 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:35.737840 31727 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.738345 30926 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.739562 30926 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:35.744519 31727 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:35.745568 30997 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:35.745707 30926 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:35.745586 30964 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:35.746701 32670 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:35.747285 32568 consensus_queue.cc:1046] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:35.747321 32670 raft_consensus.cc:695] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:35.748222 32672 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:35.748567 32672 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:35.748538 32670 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:35.749392 32672 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:35.750329 32672 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:35.759424 30819 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:35.842919 30926 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:35.844161 32621 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:35.854593 30997 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:35.855726 32617 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:36.089546 31066 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:36.090725 32654 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:36.101816 30997 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:36.102937 32583 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:36.191478 32672 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.192000 32672 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.193907 32672 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.195092 31727 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:36.195297 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.195320 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.196015 30926 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:36.196141 31722 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.196523 32612 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:36.197367 31038 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:36.198132 32672 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.198506 32672 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.198834 32672 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.209080 32672 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.210502 30926 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:36.212111 32672 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.212699 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.212049 32612 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:36.213194 30926 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.213032 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.225646 30926 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.214128 31722 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.226991 31039 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.239178 31722 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.241174 32568 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.243718 32568 raft_consensus.cc:695] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.244441 32568 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.258455 30819 catalog_manager.cc:5582] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.263142 31722 raft_consensus.cc:1215] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
I20250902 01:58:36.274690 32612 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.275120 32612 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.276496 32612 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.277192 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.277305 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.277895 31722 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.278842 30964 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:36.279357 32612 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.279692 32612 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.279987 32612 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.284740 32612 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.286131 32612 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.286674 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:36.287014 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.287489 31722 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.291482 31722 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:36.292323 30964 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:36.293004 32612 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.293579 32612 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:36.294140 32612 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.303692 30823 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.318706 32568 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.319226 32568 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.321295 32568 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.322144 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.322665 30997 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.322576 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.323758 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.324227 31722 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.325352 32568 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.325740 32568 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.326076 32568 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.330847 32568 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.333400 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.333833 30997 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.335405 32568 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.337702 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.338289 31722 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.340819 30997 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.341810 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.342430 32568 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:36.342869 32568 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.343649 32568 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.344455 31722 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.387105 32672 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.389055 32672 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.392679 32672 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.397181 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.398204 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.398689 31722 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:36.396010 32568 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.401604 32672 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.402622 32672 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.399271 30997 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.406577 32672 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.400300 32568 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.409605 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.410574 30926 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.406667 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:36.415694 30823 catalog_manager.cc:5582] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.420390 31039 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.412511 32568 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.419790 32672 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.426651 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.427570 31722 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.432067 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.433156 30995 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.433084 32672 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.435072 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.436282 31722 raft_consensus.cc:2391] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:36.424062 32568 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:36.438431 32568 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.438876 32568 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:36.440261 32672 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.440809 32672 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.441118 32672 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.441555 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.443295 31722 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.441462 32672 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.444278 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.450433 31038 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:36.453301 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.454008 32708 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.454298 30926 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.456480 32708 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.457021 32708 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.459800 32709 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.464532 32709 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.468132 32709 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.471490 32672 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.473459 32672 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.474056 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.473457 32568 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.474656 30923 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.474793 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.475605 31722 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.476356 32568 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.483119 30923 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.490720 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.491837 31727 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:36.499948 31039 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.502615 32709 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.504477 32709 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.504828 32568 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:36.504967 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.505367 32568 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.488627 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.506693 32708 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.506583 32568 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.505486 30923 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.484251 31722 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.513191 32708 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.518795 32708 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.519474 32708 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.522709 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.522897 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.523257 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.523603 30999 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.524219 31723 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.525538 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:36.526273 32709 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.526662 32709 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.527007 32709 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.528532 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.529227 31727 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.536015 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.536373 32708 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.536890 32709 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.540426 31722 raft_consensus.cc:2391] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:36.544159 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.544781 30999 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.548921 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.549455 31722 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.553048 30995 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.554425 31722 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.554463 30999 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.554761 32709 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.556429 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:36.556555 30923 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.557471 31039 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.557641 32709 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:36.558252 32708 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:36.558571 32709 raft_consensus.cc:695] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.558858 32708 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.559440 30997 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.559891 32708 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.559971 32709 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.557471 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:36.562695 32711 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.567723 31727 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:36.567996 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:36.570289 32672 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.570816 32672 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.568817 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:36.571935 32672 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.599454 32670 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.600075 32670 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.602247 32670 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.603807 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.606854 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.607846 31066 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:36.608865 30967 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:36.609637 32670 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.610193 32670 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.608834 30823 catalog_manager.cc:5582] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.610688 32670 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.614404 32654 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.615056 32654 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.622102 32670 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.620452 32612 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.623685 32612 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.625607 32612 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.626960 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:36.627169 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.627832 31075 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.629379 32654 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.630149 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.630793 31722 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:36.631796 30889 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.629913 30967 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.627487 31068 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.632620 32637 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.635491 32637 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.635887 32637 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.642701 32637 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.642999 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.643610 32717 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.644057 32717 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.644093 31066 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:36.644481 32717 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.646869 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:36.647948 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.647354 31066 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.651398 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.652057 31722 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.654853 31066 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:36.655655 32637 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.656386 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.656920 32637 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.657454 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:36.658020 31722 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.668247 31068 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:36.643512 32670 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.674470 30967 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:36.675639 32710 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.676477 32710 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.677230 32719 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:36.677963 32714 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.678670 32714 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.680881 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.681344 32711 raft_consensus.cc:695] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:36.681521 30923 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:36.682394 32711 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.683665 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.686405 31693 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:36.681031 31722 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:36.687310 32706 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:36.687757 32706 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.688087 32706 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:36.689597 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.689765 32714 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.690366 30995 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:36.691566 31038 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.685701 32637 raft_consensus.cc:695] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:36.694936 32706 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.696184 32612 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.685839 32710 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.699631 32719 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:36.701915 32612 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.702630 32719 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.684758 32717 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.704452 32612 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.696780 32714 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.709550 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:36.709673 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 49307.2us
I20250902 01:58:36.710116 31727 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.708340 32714 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.715727 32714 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.716719 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:36.715430 31722 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:36.717283 31068 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.718940 32707 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:36.718791 32612 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:36.697172 32672 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.719558 32707 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.719846 32672 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.715981 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.721079 32637 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.721679 31066 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:36.723069 30967 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.724587 32670 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:36.726943 32654 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.727690 32612 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.728451 32612 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.729350 32612 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.727581 32654 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.705056 32717 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.732414 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.733037 30923 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.733385 32672 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.734198 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.734790 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.735270 32654 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.736359 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:36.737812 32707 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.732924 32670 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.738333 32706 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.739933 32670 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.742980 31727 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:36.746750 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.747109 30924 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:36.747517 30918 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:36.747797 30964 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.747911 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.746766 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.748821 30997 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 6
I20250902 01:58:36.753412 32708 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:36.756243 31068 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:36.758911 30997 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:58:36.748749 31723 raft_consensus.cc:2366] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:58:36.760361 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.760903 30926 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:36.761335 31730 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:36.761483 32720 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:36.761855 32720 raft_consensus.cc:695] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:36.768482 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.769574 30923 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:36.751221 32707 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:36.770526 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.775508 30995 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:36.770526 31693 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.771731 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.769575 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.776731 30997 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:36.777030 32710 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.777084 32621 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.767985 32672 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20250902 01:58:36.778156 32621 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:36.778594 32621 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.777698 32710 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.784111 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.794523 30964 leader_election.cc:400] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:58:36.794792 30964 leader_election.cc:403] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:58:36.808425 30926 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:58:36.799142 32621 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.816941 32717 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.817493 32717 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.807708 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.820163 31727 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:36.820343 32621 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.821087 32656 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.815328 32708 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.826357 32708 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.827612 32710 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.828920 32708 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.821815 32724 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:58:36.819361 32717 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.825361 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.791702 32711 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20250902 01:58:36.825309 32656 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.833086 32656 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.833875 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.834518 31068 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.835953 32654 raft_consensus.cc:1062] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:58:36.835891 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.836753 30923 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:36.837798 32621 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:58:36.838032 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.838135 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.838413 32621 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:36.838917 32617 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.839600 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.839814 32617 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.840186 32617 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.840324 30918 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:36.841312 32718 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:36.841653 32718 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.841934 32718 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:36.838814 30995 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.839359 30967 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.775144 32720 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.846413 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.845222 31693 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.843668 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.849238 30924 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.838510 32654 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.810351 32707 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.853789 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.857651 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.853183 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.857834 32617 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.858446 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.860281 32710 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.852578 32717 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.861126 32724 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.866484 32711 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.882627 30997 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:36.887341 32709 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:36.871129 30822 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 4, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.889724 32710 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.890950 32709 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:36.893878 32711 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.894122 32709 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:36.896226 32711 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:36.894068 32710 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:36.904462 32724 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.905881 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:36.906190 32724 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.906658 30924 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:36.889474 32717 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.907558 32717 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:36.908154 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 93927us
I20250902 01:58:36.908299 32709 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.911804 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.911948 32583 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:36.912322 31068 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:36.912339 32583 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.913434 32617 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.914875 32583 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.915159 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.915598 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:36.918581 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 74035.8us
I20250902 01:58:36.921350 30924 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:36.898348 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 43632.7us
I20250902 01:58:36.929170 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:36.929754 30999 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:36.931335 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.933276 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:36.933511 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.924881 30995 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:36.934288 31067 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:36.934890 32621 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.937374 30824 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.923866 32709 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.959601 30967 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.961664 30889 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:36.964038 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.964560 32719 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:36.964895 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.926981 31066 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:36.967130 32670 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.963687 32637 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:36.967650 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:36.967960 30924 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:36.968343 30925 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:36.967875 30819 catalog_manager.cc:5582] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.967612 32670 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.969089 32670 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:36.969370 32722 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20250902 01:58:36.965126 32719 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:36.969588 30819 catalog_manager.cc:5582] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.971146 32719 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.968793 32714 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:36.973030 32714 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:36.974025 32710 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:36.975030 31725 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:36.971153 30819 catalog_manager.cc:5582] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.976333 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:36.980036 30923 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.982293 30819 catalog_manager.cc:5582] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:36.993852 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:36.988664 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:36.995915 32710 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.996032 31072 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:36.996582 32707 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:36.997481 30925 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:36.967975 32637 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:36.965257 32612 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.007781 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:37.010062 31066 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.010309 32729 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.010753 32729 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.012418 32732 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.012948 32732 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.014214 30923 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:37.014518 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.015017 30925 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.014391 30967 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.018368 32718 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.029086 32718 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:37.017374 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:37.031100 31001 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:37.006361 32568 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:37.032876 30926 raft_consensus.cc:2366] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.036597 32732 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.006620 32637 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.051213 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.052166 32714 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:37.052748 32714 raft_consensus.cc:695] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:37.054127 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 121847us
I20250902 01:58:37.055589 31001 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:37.037899 32568 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:37.037937 32720 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:37.060276 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.055526 31066 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.062196 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:37.062709 30997 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:37.063838 30924 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.061267 32720 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:37.069381 32717 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.068020 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.071847 31067 raft_consensus.cc:2366] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.070943 32720 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:37.072981 32717 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.078120 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.079440 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.080152 30917 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.080351 32637 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.080608 30823 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 4 to 6, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 6 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:37.081385 30920 raft_consensus.cc:2366] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:58:37.082057 32637 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.069533 30995 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:37.061393 32568 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.084607 32729 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:37.087277 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 73586.7us
I20250902 01:58:37.068033 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:37.089618 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:37.090868 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.068173 32737 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.093175 30999 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.062094 32709 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20250902 01:58:37.091329 32612 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.070421 32670 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.094599 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.100031 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.100039 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.100375 32723 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250902 01:58:37.101132 31066 raft_consensus.cc:2408] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250902 01:58:37.101411 32737 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.116338 32737 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:37.094594 31722 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:37.116940 32670 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.118484 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.119112 31066 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:37.120143 32583 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:37.121060 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:37.121282 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.121834 30924 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.121922 32733 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:37.122397 32733 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:37.123224 32733 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.077347 32672 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20250902 01:58:37.103884 32714 consensus_queue.cc:237] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.124620 31726 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:37.124641 31723 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.108986 30917 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.105103 30999 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:37.112522 30920 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.113988 30997 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:37.111387 31067 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.121533 32583 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.131899 30925 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:37.132278 30889 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:37.132901 31730 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.133002 31693 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:37.128571 30892 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:37.135073 31727 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.132030 32656 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.135931 32656 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.131850 32621 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.138165 30924 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.138427 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.140765 31039 leader_election.cc:400] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:58:37.141374 32744 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:37.141698 32744 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:37.141965 32744 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.145306 32720 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:37.146533 31001 raft_consensus.cc:2408] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250902 01:58:37.147167 32720 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.147384 32637 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.147706 32732 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:37.150063 31067 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.156975 32706 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.159816 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:37.156503 32621 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.200124 30917 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.204027 32714 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:37.204699 32714 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.206748 32714 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:37.198441 32637 raft_consensus.cc:695] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:37.186300 32672 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:37.190644 32706 raft_consensus.cc:695] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:37.221334 32732 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:37.202045 31722 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:37.225364 32708 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20250902 01:58:37.231112 32583 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.221050 32656 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.219398 31723 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:37.234709 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:37.235999 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 50578.2us
I20250902 01:58:37.238847 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.239728 30997 raft_consensus.cc:2391] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:37.232017 32672 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.241592 32732 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:37.242074 32745 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20250902 01:58:37.243090 32672 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:37.240787 31695 leader_election.cc:400] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.245146 31695 leader_election.cc:403] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:58:37.246812 30925 raft_consensus.cc:2391] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:37.247624 32621 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.248083 32583 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.248016 30965 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.256315 30920 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.258661 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:37.259604 30923 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:37.258850 30999 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250902 01:58:37.275347 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 216511us
I20250902 01:58:37.294051 31068 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.274946 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.295655 31067 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:37.296041 30999 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:37.296844 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:37.298450 32753 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:37.298838 32753 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:37.248705 32583 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.299188 32753 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.309331 31730 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:37.293413 32621 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 4
I20250902 01:58:37.311997 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:37.237661 32744 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.247979 32637 consensus_queue.cc:237] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.314051 32744 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:37.315850 31722 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEARNER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.292765 32706 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.317809 32568 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.318966 31072 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:37.320713 32568 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:37.321121 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:37.321555 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.322551 32754 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:37.322981 31722 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.324725 30925 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:37.327934 31727 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.332027 30926 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:37.335839 32718 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:37.339133 31693 leader_election.cc:400] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.340219 31693 leader_election.cc:403] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250902 01:58:37.352188 31071 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.352239 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.352893 31066 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.353281 32751 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:58:37.340749 32709 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250902 01:58:37.343312 32711 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250902 01:58:37.397801 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:37.350275 32707 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.406549 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.396633 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.413317 31071 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.356428 32672 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20250902 01:58:37.397305 32744 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20250902 01:58:37.390317 32709 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20250902 01:58:37.411583 32654 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.421252 31038 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:37.415143 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.441028 31067 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.443635 32654 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.444545 31071 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.459997 31075 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:37.463742 32759 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.464270 32759 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.414501 32763 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20250902 01:58:37.466387 31067 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:37.467417 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.424064 32732 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:37.470327 32711 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250902 01:58:37.443682 32758 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.133s
I20250902 01:58:37.444928 32751 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.474851 32751 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:37.475926 32751 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.490255 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.418884 32753 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.480489 32621 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4.
I20250902 01:58:37.466424 32759 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.522073 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.513175 31066 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:37.534076 30892 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:37.536628 32759 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.552326 32759 raft_consensus.cc:695] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:37.555395 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.556365 31071 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:37.555049 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 157339us
I20250902 01:58:37.560124 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.560783 30999 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:37.573704 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.574429 31001 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:37.556375 32759 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.579597 30920 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:58:37.564203 32654 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.584519 30997 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.585367 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:37.602922 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 470443us
I20250902 01:58:37.573278 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:37.603555 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.606277 30965 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.616307 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:37.617172 30997 raft_consensus.cc:2391] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:37.603175 32753 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.605607 32612 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.606149 32758 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:37.624276 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.618134 32737 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:37.631641 32737 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:37.632045 32737 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:37.638504 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:37.640383 30920 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:37.633203 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:37.650987 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:37.652799 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.653342 31071 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:37.622797 32754 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250902 01:58:37.628183 32758 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:37.667570 32758 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.663465 32583 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.707773 31070 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:37.712805 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:37.714459 31001 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:37.711416 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:37.649989 32670 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.715664 31073 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:37.649828 32612 raft_consensus.cc:695] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:37.729427 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:37.742637 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.739805 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:37.746673 30997 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:37.711939 32717 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.732888 32754 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.735348 32719 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:37.761157 32760 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:37.761660 32760 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.762615 32752 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20250902 01:58:37.763664 30993 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.741276 32670 raft_consensus.cc:695] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:37.767855 31001 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:37.775321 30920 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
W20250902 01:58:37.809800 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 176699us
I20250902 01:58:37.773650 32754 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:37.817929 31073 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:37.758950 32717 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.807364 32711 consensus_queue.cc:1046] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250902 01:58:37.806845 32568 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20250902 01:58:37.819262 31727 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:37.842963 306 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:37.843456 306 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:37.843847 306 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:37.818449 32612 consensus_queue.cc:237] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.900401 32754 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.839982 32670 consensus_queue.cc:237] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:37.918152 32758 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:37.930958 311 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
W20250902 01:58:37.988379 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 193520us
I20250902 01:58:37.989756 31038 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:37.993396 32760 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:37.994009 32760 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:37.994552 32760 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:37.998558 32763 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:37.999023 32763 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:37.999466 32763 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:37.991425 31038 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:38.015882 32709 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:38.019754 30823 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40172 (request call id 250) took 1014 ms. Trace:
I20250902 01:58:37.967962 32717 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:38.045156 30926 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:38.045856 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:38.049257 31075 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:38.039047 32758 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:38.191561 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 132617us
I20250902 01:58:38.202884 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:38.197491 32737 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.565s user 0.007s sys 0.003s
I20250902 01:58:38.270279 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:38.273288 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:38.277448 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:38.194674 32707 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20250902 01:58:38.214002 32760 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:38.173894 32719 raft_consensus.cc:695] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:38.213701 32709 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:38.272786 30993 raft_consensus.cc:1273] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:38.321230 30997 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:38.020097 30823 rpcz_store.cc:276] 0902 01:58:37.005532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.005672 (+ 140us) service_pool.cc:225] Handling call
0902 01:58:38.019673 (+1014001us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:37.084494 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:37.084908 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:37.084931 (+ 23us) write_op.cc:435] Acquired schema lock
0902 01:58:37.084945 (+ 14us) tablet.cc:660] Decoding operations
0902 01:58:37.085220 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:37.085267 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:37.085313 (+ 46us) tablet.cc:683] Acquiring locks for 1 operations
0902 01:58:37.085457 (+ 144us) tablet.cc:699] Row locks acquired
0902 01:58:37.085475 (+ 18us) write_op.cc:260] PREPARE: finished
0902 01:58:37.085583 (+ 108us) write_op.cc:270] Start()
0902 01:58:37.085673 (+ 90us) write_op.cc:276] Timestamp: P: 1756778317085551 usec, L: 0
0902 01:58:37.085689 (+ 16us) op_driver.cc:348] REPLICATION: starting
0902 01:58:37.295333 (+209644us) log.cc:844] Serialized 536 byte log entry
0902 01:58:37.343926 (+ 48593us) op_driver.cc:464] REPLICATION: finished
0902 01:58:37.351506 (+ 7580us) write_op.cc:301] APPLY: starting
0902 01:58:37.351600 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:37.351695 (+ 95us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:37.351800 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:37.352000 (+ 200us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:37.352134 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0902 01:58:37.352177 (+ 43us) write_op.cc:312] APPLY: finished
0902 01:58:37.352332 (+ 155us) log.cc:844] Serialized 23 byte log entry
0902 01:58:37.352484 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:37.352549 (+ 65us) write_op.cc:454] Released schema lock
0902 01:58:37.352587 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:37.414517 (+ 61930us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540026f554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212bde8 00007fc605b534e1 00007fc605b52cb8 00007fc605b57951 00007fc605b57909 00007fc605b57899 00007fc605b57861 00007fc605b56a8d 0000000000379254 0000000000379099
0902 01:58:37.416469 (+ 1952us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:37.419216 (+ 2747us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":130237,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":3355,"apply.run_cpu_time_us":1326,"apply.run_wall_time_us":1326,"mutex_wait_us":43787,"num_ops":1,"prepare.queue_time_us":1409,"prepare.run_cpu_time_us":6156,"prepare.run_wall_time_us":330180,"replication_time_us":225807,"spinlock_wait_cycles":480729472}]]}
I20250902 01:58:38.333580 30996 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250902 01:58:38.334519 316 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20250902 01:58:38.335075 30999 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
W20250902 01:58:38.336012 318 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20250902 01:58:38.321389 31073 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:38.331358 317 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:38.338945 30997 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:38.319103 31069 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250902 01:58:38.339337 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 133141us
I20250902 01:58:38.275218 30967 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:38.348419 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.025s sys 0.022s
W20250902 01:58:38.348764 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.025s sys 0.022s
I20250902 01:58:38.353931 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:38.272116 32737 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:38.317852 32759 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.050s sys 0.033s
W20250902 01:58:38.388597 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 114021us
I20250902 01:58:38.329803 32709 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.392338 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:38.399111 30967 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:38.340876 32760 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:38.341527 32711 consensus_queue.cc:1230] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20250902 01:58:38.398284 32759 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s user 0.051s sys 0.033s
I20250902 01:58:38.396421 32723 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
W20250902 01:58:38.420851 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.055s sys 0.142s
W20250902 01:58:38.421881 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.055s sys 0.142s
I20250902 01:58:38.400213 32709 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:38.451489 31073 raft_consensus.cc:3053] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:38.457628 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:38.455874 31073 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:38.344903 317 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.464937 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:38.465359 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45115.5us
I20250902 01:58:38.419173 302 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.472520 325 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:38.473031 319 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:38.473860 312 log.cc:927] Time spent T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20250902 01:58:38.474685 321 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
I20250902 01:58:38.453202 32758 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.475661 326 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.475529 30819 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:38.477425 32750 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250902 01:58:38.480542 30824 catalog_manager.cc:5582] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:58:38.454505 32760 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.481972 32763 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:38.413378 32617 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:38.484407 32568 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:38.490826 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:38.403339 32672 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250902 01:58:38.488584 31073 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:38.474213 32719 consensus_queue.cc:237] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:38.525724 319 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.526503 325 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.562995 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:38.486513 306 consensus_meta.cc:287] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.642s user 0.007s sys 0.000s
I20250902 01:58:38.525104 32749 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250902 01:58:38.490284 32717 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:38.604127 31070 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:38.489207 32723 raft_consensus.cc:695] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:38.608903 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1515) took 1101 ms. Trace:
I20250902 01:58:38.603971 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:38.505096 302 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.629074 31072 rpcz_store.cc:276] 0902 01:58:37.505142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.505326 (+ 184us) service_pool.cc:225] Handling call
0902 01:58:37.506877 (+ 1551us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:37.511507 (+ 4630us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:37.511568 (+ 61us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:37.511981 (+ 413us) log.cc:844] Serialized 26 byte log entry
0902 01:58:37.986922 (+474941us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:37.986960 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:37.987017 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:38.606170 (+619153us) raft_consensus.cc:1683] finished
0902 01:58:38.606200 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:38.606316 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8104,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":1863,"thread_start_us":466774,"threads_started":1,"wal-append.queue_time_us":682529}
I20250902 01:58:38.661267 319 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:38.657687 31071 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:38.575961 32617 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.589390 32758 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.589780 32568 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.609146 32756 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.628685 32717 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:38.608734 32583 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250902 01:58:38.626518 306 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:38.779314 31693 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:38.767755 32717 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:38.850529 30926 raft_consensus.cc:2391] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:38.773234 32737 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:38.856254 30993 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250902 01:58:38.854496 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 75287.6us
I20250902 01:58:38.774452 32756 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:38.861250 32756 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:38.862241 30996 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 4
I20250902 01:58:38.862782 32672 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.863396 32672 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.864905 32707 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.865357 32707 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.865952 32672 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:38.866390 344 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:38.866788 32656 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:38.866894 344 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.846658 32749 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:38.869367 32749 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.871938 32742 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:38.872509 32742 raft_consensus.cc:695] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:38.873533 32742 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:38.877873 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 478846us
I20250902 01:58:38.927083 30996 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250902 01:58:38.928658 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 437957us
I20250902 01:58:38.605968 326 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:38.852901 314 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:38.645411 32723 consensus_queue.cc:237] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:38.929034 32749 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:38.781810 324 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.524s
I20250902 01:58:38.986603 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:38.993515 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 428817us
W20250902 01:58:38.979674 32612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.048s sys 0.067s
I20250902 01:58:38.998004 346 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20250902 01:58:39.002890 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:39.004755 31001 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:38.979323 32672 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.013160 32763 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:39.013103 315 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20250902 01:58:38.992025 317 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:39.004086 32612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.048s sys 0.067s
W20250902 01:58:39.043649 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.009s sys 0.031s
W20250902 01:58:39.044078 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.009s sys 0.031s
W20250902 01:58:39.002786 31721 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.018s sys 0.051s
I20250902 01:58:39.064034 305 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:38.844533 31722 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:58:39.009374 314 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.150942 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:39.518744 337 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.410s user 0.001s sys 0.000s
W20250902 01:58:39.149084 31721 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s user 0.018s sys 0.052s
I20250902 01:58:39.323453 32568 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.268754 32760 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.327286 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:39.348120 325 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:39.586982 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.004s sys 0.008s
W20250902 01:58:39.596727 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.004s sys 0.008s
W20250902 01:58:39.597186 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 612885us
W20250902 01:58:39.598186 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:39.177640 345 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:39.177425 306 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:39.178304 32617 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:39.286706 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 428842us
I20250902 01:58:39.234422 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:39.177758 336 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.345s user 0.000s sys 0.001s
I20250902 01:58:39.549201 31722 raft_consensus.cc:604] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:39.384027 329 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:39.048345 32670 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.327265 322 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:39.048617 319 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:39.551045 32568 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:39.476457 350 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20250902 01:58:39.599129 31001 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.590s user 0.012s sys 0.001s
I20250902 01:58:39.658067 31001 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:58:39.390120 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:39.016348 32672 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.375329 32755 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.373930 32758 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.682632 317 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:39.698078 313 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.691071 32760 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:39.691383 322 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.655807 32612 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:39.659569 32752 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.700758 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 746) took 1669 ms. Trace:
I20250902 01:58:39.354311 303 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.682791 32708 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.715519 32758 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:39.725292 32709 consensus_meta.cc:287] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.231s user 0.009s sys 0.000s
I20250902 01:58:39.725904 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 492) took 1668 ms. Trace:
I20250902 01:58:39.726089 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1526) took 1329 ms. Trace:
W20250902 01:58:39.732278 31075 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.017s sys 0.037s
W20250902 01:58:39.825632 31075 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.018s sys 0.037s
I20250902 01:58:39.586149 305 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.657436 32670 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:39.732503 31692 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.010s sys 0.043s
W20250902 01:58:39.746785 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.004s sys 0.047s
W20250902 01:58:39.840830 31692 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.862s user 0.011s sys 0.043s
W20250902 01:58:39.840927 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.006s sys 0.049s
W20250902 01:58:39.841252 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 896695us
W20250902 01:58:39.841359 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 828275us
W20250902 01:58:39.712956 32656 consensus_meta.cc:287] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.846s user 0.010s sys 0.000s
W20250902 01:58:39.762077 30918 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.017s sys 0.029s
W20250902 01:58:39.851457 30918 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.838s user 0.017s sys 0.029s
W20250902 01:58:39.714723 339 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.862s user 0.002s sys 0.012s
I20250902 01:58:39.715551 317 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:39.820447 32717 consensus_meta.cc:287] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.892s user 0.008s sys 0.000s
W20250902 01:58:39.747406 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.026s sys 0.020s
W20250902 01:58:39.859977 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.014s sys 0.010s
I20250902 01:58:39.716045 326 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:39.860380 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s user 0.029s sys 0.020s
W20250902 01:58:39.860905 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 857736us
W20250902 01:58:39.856338 30965 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.613s (SENT) [suppressed 18 similar messages]
W20250902 01:58:39.861498 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.014s sys 0.011s
I20250902 01:58:39.660548 319 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.661116 345 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:39.863670 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.000s sys 0.055s
W20250902 01:58:39.863992 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.000s sys 0.055s
W20250902 01:58:39.864289 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 815921us
I20250902 01:58:39.824721 32708 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.855824 314 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.761456 32755 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.856864 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:39.763442 32729 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.689934 32568 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.820825 313 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.853196 32656 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:39.860517 32711 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.932885 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:39.864557 325 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.839967 32709 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.701536 329 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.763119 32612 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.760810 32760 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.935369 31072 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:39.851605 32758 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.862241 32717 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.824980 32752 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.894411 370 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:39.981375 313 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.982177 373 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.982537 376 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.983805 31068 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:39.986588 30999 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:39.901443 345 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:39.947898 353 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.990667 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:39.933075 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1661) took 1656 ms. Trace:
I20250902 01:58:39.991232 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 289) took 1943 ms. Trace:
I20250902 01:58:39.953953 32720 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:39.979836 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:39.992671 30917 rpcz_store.cc:276] 0902 01:58:38.043850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.043971 (+ 121us) service_pool.cc:225] Handling call
0902 01:58:38.168175 (+124204us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.193558 (+ 25383us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.193598 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.205138 (+ 11540us) log.cc:844] Serialized 26 byte log entry
0902 01:58:38.500734 (+295596us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:38.500783 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:38.500860 (+ 77us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:39.712588 (+1211728us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:39.973899 (+261311us) raft_consensus.cc:1683] finished
0902 01:58:39.973923 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:39.987267 (+ 13344us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":25120,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":13573,"spinlock_wait_cycles":9857408,"thread_start_us":224760,"threads_started":1,"wal-append.queue_time_us":447101}
I20250902 01:58:39.978639 31069 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:39.934929 32729 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.986958 32656 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.934863 314 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20250902 01:58:39.982668 373 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.856787 303 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.825151 31066 rpcz_store.cc:276] 0902 01:58:38.397119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.400033 (+ 2914us) service_pool.cc:225] Handling call
0902 01:58:39.709173 (+1309140us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff1915ce 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:39.713129 (+ 3956us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.725629 (+ 12500us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:39.725672 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:39.725745 (+ 73us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.725784 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:39.725862 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:39.726025 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57011456}
I20250902 01:58:39.893601 359 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:39.972085 32711 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:39.954973 325 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.024019 32709 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:39.818473 31727 rpcz_store.cc:276] 0902 01:58:38.031778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.032827 (+ 1049us) service_pool.cc:225] Handling call
0902 01:58:38.048200 (+ 15373us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.274860 (+226660us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.278823 (+ 3963us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.530232 (+251409us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b2c000a5ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:58:38.530623 (+ 391us) log.cc:844] Serialized 26 byte log entry
0902 01:58:39.040433 (+509810us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.047584 (+ 7151us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:39.151572 (+103988us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:39.700376 (+548804us) raft_consensus.cc:1683] finished
0902 01:58:39.700406 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:39.700660 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5087,"raft.queue_time_us":70316,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":43163,"spinlock_wait_cycles":5378432,"thread_start_us":463835,"threads_started":1,"wal-append.queue_time_us":1082086}
I20250902 01:58:40.007040 32672 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.024740 32729 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:39.998101 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:39.998183 305 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.998360 32706 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:39.991663 370 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.825281 31721 rpcz_store.cc:276] 0902 01:58:38.057703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.212143 (+154440us) service_pool.cc:225] Handling call
0902 01:58:38.331071 (+118928us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.406659 (+ 75588us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.411694 (+ 5035us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.465452 (+ 53758us) log.cc:844] Serialized 26 byte log entry
0902 01:58:39.523075 (+1057623us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.523115 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:39.523187 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:39.725651 (+202464us) raft_consensus.cc:1683] finished
0902 01:58:39.725682 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:39.725839 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13716,"raft.queue_time_us":73470,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":44684,"thread_start_us":1057197,"threads_started":1,"wal-append.queue_time_us":1158250}
I20250902 01:58:40.048274 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 379) took 1354 ms. Trace:
I20250902 01:58:40.032909 368 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.025803 32656 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.053416 31068 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:40.053854 30996 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:40.057268 373 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:39.998968 32720 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:39.982955 376 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:40.068414 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:40.049470 325 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.032141 32708 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.069038 31075 rpcz_store.cc:276] 0902 01:58:38.693991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.695397 (+ 1406us) service_pool.cc:225] Handling call
0902 01:58:38.771674 (+ 76277us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.780166 (+ 8492us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.780206 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.790089 (+ 9883us) log.cc:844] Serialized 26 byte log entry
0902 01:58:39.826064 (+1035975us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.826119 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:39.826181 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.048077 (+221896us) raft_consensus.cc:1683] finished
0902 01:58:40.048098 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.048222 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":84,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8237,"thread_start_us":822242,"threads_started":1,"wal-append.queue_time_us":903600}
I20250902 01:58:40.062873 315 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.064292 32670 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.020054 353 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.063318 317 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.078078 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:40.078586 365 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20250902 01:58:40.081867 32717 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.066756 32706 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.085376 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 495) took 1756 ms. Trace:
I20250902 01:58:40.090181 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.088641 31730 rpcz_store.cc:276] 0902 01:58:38.328647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.348966 (+ 20319us) service_pool.cc:225] Handling call
0902 01:58:40.084652 (+1735686us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:40.084850 (+ 198us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:40.085039 (+ 189us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:40.085072 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.085119 (+ 47us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:40.085165 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.085223 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.085320 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7347968}
I20250902 01:58:40.075884 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.097712 356 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:40.098174 32755 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:40.098776 361 log.cc:927] Time spent T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20250902 01:58:40.064890 359 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.084450 32612 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.104270 32706 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.087842 32568 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.094743 32758 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.115505 322 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.098457 356 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.106498 32760 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.101748 32752 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:40.122139 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 469134us
I20250902 01:58:40.084875 368 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:40.131088 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 274787us
I20250902 01:58:40.138957 368 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.140298 359 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.140975 378 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.141527 378 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:40.146078 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 198331us
I20250902 01:58:40.095105 315 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.104485 353 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.136265 32707 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.155274 32711 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.131182 329 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.131721 370 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:39.996461 30992 rpcz_store.cc:276] 0902 01:58:38.211975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.328234 (+116259us) service_pool.cc:225] Handling call
0902 01:58:38.331354 (+ 3120us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.396822 (+ 65468us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.399848 (+ 3026us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.464613 (+ 64765us) log.cc:844] Serialized 26 byte log entry
0902 01:58:38.978873 (+514260us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:38.982818 (+ 3945us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:38.991728 (+ 8910us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:39.861617 (+869889us) raft_consensus.cc:1683] finished
0902 01:58:39.863093 (+ 1476us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:39.868483 (+ 5390us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58901,"raft.queue_time_us":14777,"raft.run_cpu_time_us":8387,"raft.run_wall_time_us":54070,"thread_start_us":462933,"threads_started":1,"wal-append.queue_time_us":587024}
I20250902 01:58:40.158336 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1667) took 1146 ms. Trace:
I20250902 01:58:40.166630 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.181701 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.171402 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.195998 30917 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:40.180749 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:40.192209 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45487.5us
W20250902 01:58:40.199005 31041 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.035s sys 0.042s
W20250902 01:58:40.206081 31041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s user 0.035s sys 0.042s
W20250902 01:58:40.210827 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 337380us
I20250902 01:58:40.192634 30918 rpcz_store.cc:276] 0902 01:58:39.011603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.011705 (+ 102us) service_pool.cc:225] Handling call
0902 01:58:39.013535 (+ 1830us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.013692 (+ 157us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:39.856492 (+842800us) log.cc:844] Serialized 14 byte log entry
0902 01:58:40.158037 (+301545us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.158083 (+ 46us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:40.158119 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.158176 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.158282 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":966425,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":848,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":1138127,"threads_started":2}
I20250902 01:58:40.215008 310 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.165364 32656 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.165482 32672 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.179409 32720 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.178409 353 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.227807 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.215724 310 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.157284 32568 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.171201 356 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:40.229625 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:40.224701 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.243419 31730 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:40.233824 329 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.171434 378 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.231451 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.231514 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.230019 31692 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:40.229907 32754 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.206274 376 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.216444 32707 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:40.156860 32723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.008s sys 0.045s
W20250902 01:58:40.211758 375 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20250902 01:58:40.222518 32729 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:40.247869 30925 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:40.249377 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1660) took 1980 ms. Trace:
I20250902 01:58:40.241449 32758 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.252740 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 383) took 1557 ms. Trace:
I20250902 01:58:40.244099 329 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.244755 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.255136 32707 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.269515 31000 rpcz_store.cc:276] 0902 01:58:38.199512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.318472 (+118960us) service_pool.cc:225] Handling call
0902 01:58:38.349549 (+ 31077us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:38.738165 (+388616us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.738999 (+ 834us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:38.775334 (+ 36335us) log.cc:844] Serialized 26 byte log entry
0902 01:58:39.299774 (+524440us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.299853 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:39.299938 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.179814 (+879876us) raft_consensus.cc:1683] finished
0902 01:58:40.179841 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.179990 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51871,"raft.queue_time_us":121856,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":667578,"threads_started":2,"wal-append.queue_time_us":1008145}
I20250902 01:58:40.237799 353 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.118780 32744 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:40.234138 32711 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.262348 30892 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:40.266158 31692 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:40.268409 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 496) took 1002 ms. Trace:
I20250902 01:58:40.270933 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.255888 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1764) took 1248 ms. Trace:
I20250902 01:58:40.269789 31074 rpcz_store.cc:276] 0902 01:58:38.695233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.778444 (+ 83211us) service_pool.cc:225] Handling call
0902 01:58:39.621873 (+843429us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:39.622128 (+ 255us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.972505 (+350377us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:39.972544 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:39.972591 (+ 47us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:39.972630 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.252400 (+279770us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b540024c7d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:40.252449 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.252560 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":809753856}
W20250902 01:58:40.264359 32723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.008s sys 0.046s
I20250902 01:58:40.434573 31075 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:40.245776 32568 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.244712 315 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.433099 31729 rpcz_store.cc:276] 0902 01:58:39.260811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.260956 (+ 145us) service_pool.cc:225] Handling call
0902 01:58:39.530905 (+269949us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.991035 (+460130us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:39.991065 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:39.991103 (+ 38us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:39.991133 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.256736 (+265603us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b540054d3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:40.256776 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.263294 (+ 6518us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45708,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":110,"spinlock_wait_cycles":1029854592}
I20250902 01:58:40.269495 356 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.237366 32672 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:40.460114 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 216446us
I20250902 01:58:40.257279 32754 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.435184 30999 rpcz_store.cc:276] 0902 01:58:39.007822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.007977 (+ 155us) service_pool.cc:225] Handling call
0902 01:58:39.016292 (+ 8315us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.986974 (+970682us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.255656 (+268682us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54000d70d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:40.255839 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2231450880}
I20250902 01:58:40.279992 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.476317 372 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:40.461493 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.030s sys 0.027s
W20250902 01:58:40.496760 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.030s sys 0.027s
I20250902 01:58:40.474740 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.504604 310 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.505599 31067 raft_consensus.cc:2408] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250902 01:58:40.511315 31001 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.025s sys 0.043s
W20250902 01:58:40.512727 31001 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.507s user 0.025s sys 0.043s
W20250902 01:58:40.512899 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 246866us
I20250902 01:58:40.439666 376 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:40.504662 335 log.cc:927] Time spent T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 1.822s user 0.002s sys 0.021s
I20250902 01:58:40.354126 31727 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:40.453135 379 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.277078 32758 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.520706 310 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:40.522449 378 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.541710 32720 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.534276 315 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.477396 356 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.546540 32718 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:40.614302 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 269286us
I20250902 01:58:40.541855 32708 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.524039 32656 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20250902 01:58:40.481957 32711 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.621206 347 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.553831 390 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20250902 01:58:40.525269 376 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.637503 329 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.637699 376 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.556162 314 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:40.634289 31693 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:40.634492 353 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.649757 376 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.650331 376 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.652329 376 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.649855 329 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.653681 31693 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:40.656183 31073 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:40.652917 329 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:40.661761 328 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 2.228s user 0.010s sys 0.072s
I20250902 01:58:40.664816 32728 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:40.665503 32672 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.665731 32728 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.666128 32728 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:40.666188 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 745) took 2838 ms. Trace:
I20250902 01:58:40.638705 32708 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.673782 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.276978 32729 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.638228 314 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.666661 31726 rpcz_store.cc:276] 0902 01:58:37.827355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.831945 (+ 4590us) service_pool.cc:225] Handling call
0902 01:58:37.849306 (+ 17361us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:37.886835 (+ 37529us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:37.886879 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:37.894044 (+ 7165us) log.cc:844] Serialized 26 byte log entry
0902 01:58:38.033026 (+138982us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:38.033098 (+ 72us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:38.033180 (+ 82us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.275399 (+2242219us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:40.663821 (+388422us) raft_consensus.cc:1683] finished
0902 01:58:40.663847 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.666099 (+ 2252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8546,"raft.queue_time_us":32949,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6142,"spinlock_wait_cycles":17811840,"thread_start_us":119771,"threads_started":1,"wal-append.queue_time_us":448402}
I20250902 01:58:40.675236 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:40.654167 32656 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:40.568892 372 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.689846 372 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.627681 378 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.690786 31000 raft_consensus.cc:2366] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:58:40.630838 32718 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.626952 315 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.635597 32720 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.675928 359 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.686561 314 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:40.659191 347 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.724300 329 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.691936 32656 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.567909 379 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.728153 32754 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.718906 32707 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.716161 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.743050 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.744444 315 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.728889 32656 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:40.730000 32568 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.724926 359 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.742375 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 743) took 2934 ms. Trace:
I20250902 01:58:40.765619 32758 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.791677 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1765) took 1776 ms. Trace:
I20250902 01:58:40.796157 379 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.797964 347 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.788379 32708 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.800319 31073 raft_consensus.cc:1273] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250902 01:58:40.839617 31073 pending_rounds.cc:85] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 1
I20250902 01:58:40.804431 378 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.805110 32728 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.791090 32711 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.815409 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 388) took 1800 ms. Trace:
I20250902 01:58:40.816761 314 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.820058 32720 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.820676 32718 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.821766 359 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.767644 30999 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:40.827935 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 565824us
W20250902 01:58:40.827176 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 358761us
W20250902 01:58:40.829196 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 301032us
W20250902 01:58:40.833068 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 354982us
I20250902 01:58:40.881589 31692 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:40.824855 344 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.859233 31071 rpcz_store.cc:276] 0902 01:58:39.014933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.694027 (+679094us) service_pool.cc:225] Handling call
0902 01:58:39.694418 (+ 391us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.995889 (+301471us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:39.999226 (+ 3337us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.004473 (+ 5247us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:40.006827 (+ 2354us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.815165 (+808338us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400157054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:40.815211 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.815355 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":675372288}
I20250902 01:58:40.874811 32728 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.876420 314 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:40.877036 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.883977 30994 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:40.876974 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.884361 32711 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.884783 32711 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.884886 31038 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:40.885195 32708 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.885685 32708 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.885754 32656 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.886494 32711 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.878701 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.888074 32656 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.888492 31000 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:40.888911 32711 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:40.889257 32708 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.889369 32711 raft_consensus.cc:695] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:40.889966 344 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:40.890208 32711 consensus_queue.cc:237] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.891140 31038 leader_election.cc:400] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:58:40.891528 31038 leader_election.cc:403] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250902 01:58:40.892172 396 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.892488 376 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
I20250902 01:58:40.892607 396 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.893273 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:40.894299 388 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:40.894392 396 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:40.894708 388 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:40.894503 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:40.895130 388 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:40.895504 329 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:40.895900 329 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:40.896240 329 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:40.899030 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.900274 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.900907 31071 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:40.827666 31727 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:40.839886 31073 pending_rounds.cc:107] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250902 01:58:40.902420 31073 raft_consensus.cc:2887] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:40.903152 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1522) took 2512 ms. Trace:
I20250902 01:58:40.903437 31073 rpcz_store.cc:276] 0902 01:58:38.390951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.401654 (+ 10703us) service_pool.cc:225] Handling call
0902 01:58:38.451379 (+ 49725us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:40.655722 (+2204343us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b580069efb0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020ab003 00007fc60210e577 00007fc602126fa7 00007fc60212879e 00007fc602132af9 00007fc6021333c6 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44
0902 01:58:40.902867 (+247145us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.903057 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36114816}
I20250902 01:58:40.826602 32724 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.905189 32724 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.905907 31730 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:40.904302 32709 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:40.907495 32724 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.907311 31074 raft_consensus.cc:1273] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:40.910336 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.912120 32709 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:40.912667 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.912138 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:40.913555 32751 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250902 01:58:40.914005 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.914680 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.914951 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.916275 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.917428 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.918354 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.919014 31725 raft_consensus.cc:1238] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:40.919381 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.920369 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.916949 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:40.854406 326 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.072s sys 0.128s
I20250902 01:58:40.922772 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:40.920370 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.921675 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:40.927981 31069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.009s sys 0.009s
W20250902 01:58:40.930491 31069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s user 0.009s sys 0.009s
W20250902 01:58:40.829663 327 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 1.974s user 0.023s sys 0.029s
W20250902 01:58:40.936620 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 275861us
I20250902 01:58:40.941978 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:40.927193 326 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.072s sys 0.129s
I20250902 01:58:40.908233 31727 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:40.946475 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.945619 30892 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:40.949611 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.949743 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.950906 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1530) took 1064 ms. Trace:
I20250902 01:58:40.951920 30924 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:40.954027 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.922094 382 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:40.954358 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:40.959338 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:40.952904 31000 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:40.954562 31729 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:40.959926 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.961732 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.962244 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.962422 31729 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:40.963091 31721 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:40.963729 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:40.965078 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.965633 31729 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:40.965876 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.966364 31728 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:40.966317 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.967293 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.967862 31728 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:40.968017 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.968288 349 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.968684 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.968921 349 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.970410 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.970961 349 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.971143 31721 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:40.972565 349 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:40.967981 32720 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:40.973484 32720 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.976123 32720 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.977185 349 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:40.979382 349 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:40.980001 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:40.984407 30964 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:40.999389 30917 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:58:41.001055 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.001626 30999 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
I20250902 01:58:41.003400 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.002621 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.004113 30967 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:40.946269 396 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:41.006594 30964 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:41.007547 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.007730 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.009372 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:41.009676 31695 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:41.009941 30919 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:40.831595 30993 rpcz_store.cc:276] 0902 01:58:39.015205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.015716 (+ 511us) service_pool.cc:225] Handling call
0902 01:58:39.178915 (+163199us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:39.545180 (+366265us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:39.584626 (+ 39446us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:39.697319 (+112693us) log.cc:844] Serialized 26 byte log entry
0902 01:58:40.145111 (+447792us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:40.150469 (+ 5358us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.158151 (+ 7682us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.787355 (+629204us) raft_consensus.cc:1683] finished
0902 01:58:40.788051 (+ 696us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.790851 (+ 2800us) inbound_call.cc:173] Queueing success response
0902 01:58:40.825937 (+ 35086us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b48001503d8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
Metrics: {"mutex_wait_us":63585,"raft.queue_time_us":431110,"raft.run_cpu_time_us":20811,"raft.run_wall_time_us":1201500,"spinlock_wait_cycles":15114496,"thread_start_us":392251,"threads_started":1,"wal-append.queue_time_us":920894}
I20250902 01:58:41.004460 31074 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:41.008492 31000 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:41.015062 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.028563 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.029340 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:41.034256 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.036062 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:41.036126 31074 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250902 01:58:41.036942 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 91424.2us
I20250902 01:58:41.037541 31074 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:40.998764 376 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:41.034333 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.039484 31038 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:41.007295 349 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:41.023903 344 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:41.028488 32718 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:41.012353 314 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:41.042625 30964 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:41.030066 31725 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:41.052313 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.056638 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.057102 31729 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 1.
I20250902 01:58:41.057215 31073 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:41.040719 310 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:41.007205 32720 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250902 01:58:41.037688 329 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:41.028060 32729 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:41.025210 326 consensus_queue.cc:1046] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20250902 01:58:41.060372 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.034967 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.029904 396 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:41.053119 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.041780 378 consensus_queue.cc:1046] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250902 01:58:40.826272 31724 rpcz_store.cc:276] 0902 01:58:37.805421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.806707 (+ 1286us) service_pool.cc:225] Handling call
0902 01:58:37.818245 (+ 11538us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:37.850512 (+ 32267us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:37.851634 (+ 1122us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:37.852189 (+ 555us) log.cc:844] Serialized 26 byte log entry
0902 01:58:38.044363 (+192174us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:38.050126 (+ 5763us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:38.145948 (+ 95822us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.622124 (+2476176us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:40.716495 (+ 94371us) raft_consensus.cc:1683] finished
0902 01:58:40.719099 (+ 2604us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.739869 (+ 20770us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13210,"raft.queue_time_us":11112,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":34151,"spinlock_wait_cycles":151985024,"thread_start_us":181905,"threads_started":1,"wal-append.queue_time_us":424880}
I20250902 01:58:41.180188 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 256) took 2215 ms. Trace:
I20250902 01:58:41.180958 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1669) took 1234 ms. Trace:
I20250902 01:58:41.345516 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:41.289732 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:41.280625 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 332087us
I20250902 01:58:41.096089 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.292348 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:41.281438 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 277387us
I20250902 01:58:41.294318 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:41.059641 375 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.009s sys 0.025s
I20250902 01:58:41.053004 32718 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:41.344816 407 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20250902 01:58:41.341094 410 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20250902 01:58:41.399852 30967 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:41.344728 319 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:41.405174 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 365859us
I20250902 01:58:41.355331 30996 rpcz_store.cc:276] 0902 01:58:39.946938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.091215 (+144277us) service_pool.cc:225] Handling call
0902 01:58:40.091495 (+ 280us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:40.109645 (+ 18150us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:40.109682 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.109729 (+ 47us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:40.109768 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:41.180668 (+1070900us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5400177d54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:41.180726 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:41.180899 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41273088}
I20250902 01:58:41.401140 30994 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:41.160781 327 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20250902 01:58:41.289785 348 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:41.096331 32711 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:41.177628 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.345531 324 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:41.125108 372 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:41.485838 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.354110 30998 rpcz_store.cc:276] 0902 01:58:38.770494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.773123 (+ 2629us) service_pool.cc:225] Handling call
0902 01:58:40.985343 (+2212220us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:40.985539 (+ 196us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:40.985714 (+ 175us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:40.985745 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.985800 (+ 55us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:40.985836 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.985895 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.985999 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6245248}
I20250902 01:58:41.486008 372 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:41.486761 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.180087 314 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:41.290808 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1664) took 1357 ms. Trace:
I20250902 01:58:41.291115 32568 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:41.426716 31038 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:41.486761 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.096452 349 consensus_queue.cc:1046] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.371s
I20250902 01:58:41.431427 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.355849 32754 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:41.096606 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 742) took 3452 ms. Trace:
I20250902 01:58:41.464910 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.465441 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.541106 30996 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:41.541924 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.094035 310 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:41.465723 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.548019 31073 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:41.166349 32706 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:41.053278 376 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:41.402470 32758 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:41.418082 32718 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:41.344169 388 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:41.480031 324 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:41.092561 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:41.404232 375 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.871s user 0.013s sys 0.029s
I20250902 01:58:41.401149 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.426652 378 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:41.488099 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.091863 400 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.413s
I20250902 01:58:41.430941 32719 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:41.536618 31000 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:41.581993 310 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:41.096871 332 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:41.581518 31726 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:41.583864 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:41.479099 30992 raft_consensus.cc:2391] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:41.475805 32711 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:41.481433 319 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:41.618408 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:41.586476 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.537735 31071 raft_consensus.cc:1238] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250902 01:58:41.662478 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.665179 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.540148 30924 raft_consensus.cc:1238] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:41.549392 32754 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:41.550173 32568 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:41.669615 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:41.961637 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:41.961941 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 248) took 3470 ms. Trace:
I20250902 01:58:41.743499 31074 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:42.035923 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 276) took 1053 ms. Trace:
I20250902 01:58:41.974191 31073 raft_consensus.cc:2408] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:58:41.741400 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.046061 31000 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:42.047292 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.043097 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:42.009454 30997 rpcz_store.cc:276] 0902 01:58:38.491502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.491630 (+ 128us) service_pool.cc:225] Handling call
0902 01:58:38.531320 (+ 39690us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:38.607620 (+ 76300us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.607658 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:38.607700 (+ 42us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:38.607730 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:41.961718 (+3353988us) spinlock_profiling.cc:243] Waited 76.1 ms on lock 0x7b54000b5754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:41.961758 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:41.961891 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175069952}
I20250902 01:58:42.043336 30996 rpcz_store.cc:276] 0902 01:58:40.982709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.618111 (+635402us) service_pool.cc:225] Handling call
0902 01:58:42.035856 (+417745us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.049209 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:42.049847 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.050207 31000 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
W20250902 01:58:41.894852 31070 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1486) took 3667 ms (client timeout 1613 ms). Trace:
I20250902 01:58:42.055527 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1545) took 1013 ms. Trace:
I20250902 01:58:42.056048 31073 rpcz_store.cc:276] 0902 01:58:41.042386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.586934 (+544548us) service_pool.cc:225] Handling call
0902 01:58:42.055446 (+468512us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:41.960075 415 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.479s user 0.000s sys 0.000s
I20250902 01:58:42.058254 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1491) took 1157 ms. Trace:
W20250902 01:58:41.960736 409 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.552s user 0.001s sys 0.000s
I20250902 01:58:42.066496 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:42.052848 31070 rpcz_store.cc:269] 0902 01:58:38.202555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.202675 (+ 120us) service_pool.cc:225] Handling call
0902 01:58:41.869278 (+3666603us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400256f54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:41.869419 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921381760}
W20250902 01:58:42.048981 31068 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1525) took 3657 ms (client timeout 2645 ms). Trace:
I20250902 01:58:42.062767 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.070086 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:42.067880 31068 rpcz_store.cc:269] 0902 01:58:38.391816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.392090 (+ 274us) service_pool.cc:225] Handling call
0902 01:58:42.048819 (+3656729us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400259754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:42.048916 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3648716032}
I20250902 01:58:41.662153 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:42.064008 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:41.959431 403 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.487s user 0.000s sys 0.000s
I20250902 01:58:41.660589 324 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:42.046248 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 274) took 1038 ms. Trace:
I20250902 01:58:42.074563 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:41.818915 412 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:41.818378 32708 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:41.664327 378 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:42.069432 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.008s user 0.076s sys 0.125s
I20250902 01:58:41.821859 348 raft_consensus.cc:3053] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:41.961905 372 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:42.081329 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.017s sys 0.070s
W20250902 01:58:42.123064 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s user 0.018s sys 0.070s
I20250902 01:58:41.756297 349 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:42.083274 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.667876 32719 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:41.753611 420 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:41.754912 396 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.084376 30925 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:41.660493 32720 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.446s
I20250902 01:58:42.044921 32711 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:41.617779 32706 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.072489 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 390) took 1158 ms. Trace:
I20250902 01:58:42.063447 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:42.146067 31041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s user 0.014s sys 0.054s
I20250902 01:58:42.146363 30994 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:42.146886 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 264) took 1171 ms. Trace:
I20250902 01:58:42.045042 319 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:42.113188 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:42.112929 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.094130 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 389) took 1180 ms. Trace:
I20250902 01:58:42.145665 31068 rpcz_store.cc:276] 0902 01:58:40.913877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.071670 (+1157793us) service_pool.cc:225] Handling call
0902 01:58:42.071966 (+ 296us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.072144 (+ 178us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.072178 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.072238 (+ 60us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:42.072274 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.072330 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.072436 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:41.996749 332 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:41.792300 424 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:42.119709 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:41.975747 314 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:42.121026 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.066s user 0.076s sys 0.125s
I20250902 01:58:42.075935 329 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.127987 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:41.620788 32758 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.082159 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1679) took 1129 ms. Trace:
I20250902 01:58:42.086766 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.167191 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.167742 30996 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:42.168550 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.136211 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1507) took 1223 ms. Trace:
I20250902 01:58:42.148557 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:42.148105 31070 rpcz_store.cc:276] 0902 01:58:40.913563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.069828 (+1156265us) service_pool.cc:225] Handling call
0902 01:58:42.093638 (+ 23810us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.093807 (+ 169us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.093838 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.093882 (+ 44us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:42.093918 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.093975 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.094074 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.147225 30994 rpcz_store.cc:276] 0902 01:58:40.975686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.742729 (+767043us) service_pool.cc:225] Handling call
0902 01:58:42.146837 (+404108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.148162 30997 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:42.121018 30924 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:42.121328 32708 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:42.148955 430 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:42.150210 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 188051us
I20250902 01:58:42.159807 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 293) took 3807 ms. Trace:
W20250902 01:58:42.159809 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.11733e+06us
I20250902 01:58:42.129400 32719 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:41.536427 30995 rpcz_store.cc:276] 0902 01:58:39.933634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.953737 (+ 20103us) service_pool.cc:225] Handling call
0902 01:58:41.290164 (+1336427us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:41.290343 (+ 179us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:41.290494 (+ 151us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:41.290521 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:41.290559 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:41.290590 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:41.290641 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:41.290748 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7489920}
I20250902 01:58:42.161839 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.122727 348 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:42.087060 30993 rpcz_store.cc:276] 0902 01:58:40.982114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.035829 (+ 53715us) service_pool.cc:225] Handling call
0902 01:58:42.020099 (+984270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.169528 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1682) took 1215 ms. Trace:
I20250902 01:58:42.196448 30997 rpcz_store.cc:276] 0902 01:58:40.953917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.064858 (+1110941us) service_pool.cc:225] Handling call
0902 01:58:42.169480 (+104622us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.169111 31073 rpcz_store.cc:276] 0902 01:58:40.912449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.086495 (+1174046us) service_pool.cc:225] Handling call
0902 01:58:42.124904 (+ 38409us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.135776 (+ 10872us) raft_consensus.cc:1532] Early marking committed up to index 1
0902 01:58:42.135817 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.135871 (+ 54us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:42.135913 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.135988 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.136133 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.158759 332 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:42.157727 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 391) took 1241 ms. Trace:
I20250902 01:58:42.148912 319 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:42.171308 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.176167 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.180859 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.149820 396 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:42.199766 31068 rpcz_store.cc:276] 0902 01:58:40.916692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.156814 (+1240122us) service_pool.cc:225] Handling call
0902 01:58:42.157105 (+ 291us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.157321 (+ 216us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.157393 (+ 72us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.157447 (+ 54us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:42.157488 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.157556 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.157666 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.180182 30920 rpcz_store.cc:276] 0902 01:58:38.345730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.345873 (+ 143us) service_pool.cc:225] Handling call
0902 01:58:38.346187 (+ 314us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:38.641871 (+295684us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:38.641923 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:38.641978 (+ 55us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:38.642032 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.152895 (+3510863us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b54000604d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:42.152934 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.153034 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679671680}
W20250902 01:58:42.201268 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1517) took 3853 ms (client timeout 1613 ms). Trace:
I20250902 01:58:42.201474 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 397) took 1183 ms. Trace:
I20250902 01:58:42.168737 376 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.121479 412 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.166513 30992 rpcz_store.cc:276] 0902 01:58:40.942826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.943869 (+ 1043us) service_pool.cc:225] Handling call
0902 01:58:42.066980 (+1123111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792}
I20250902 01:58:42.162060 425 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:42.195081 30918 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:42.216392 30964 consensus_peers.cc:489] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92f16b8a15144fcf8807608bdee71123. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:42.142426 349 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.199015 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:42.223019 30994 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:42.201642 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.058645 31067 rpcz_store.cc:276] 0902 01:58:40.900723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.901357 (+ 634us) service_pool.cc:225] Handling call
0902 01:58:40.901729 (+ 372us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:40.902111 (+ 382us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:40.902153 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:40.902689 (+ 536us) log.cc:844] Serialized 26 byte log entry
0902 01:58:41.060186 (+157497us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:41.096373 (+ 36187us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:41.125479 (+ 29106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:42.058020 (+932541us) raft_consensus.cc:1683] finished
0902 01:58:42.058044 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.058195 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1458,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":150918,"threads_started":1,"wal-append.queue_time_us":437664}
I20250902 01:58:42.224337 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.146229 420 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.194670 30925 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:42.164824 31726 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:42.201848 31068 rpcz_store.cc:276] 0902 01:58:41.017964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.200825 (+1182861us) service_pool.cc:225] Handling call
0902 01:58:42.201067 (+ 242us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.201197 (+ 130us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.201217 (+ 20us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.201257 (+ 40us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:42.201281 (+ 24us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.201324 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.201415 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.165210 424 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.204254 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 396) took 1286 ms. Trace:
I20250902 01:58:42.214181 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.204895 433 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:42.204905 416 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.421s user 0.001s sys 0.000s
I20250902 01:58:42.205888 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:42.169917 378 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.258220 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 271) took 1277 ms. Trace:
I20250902 01:58:42.258674 30996 rpcz_store.cc:276] 0902 01:58:40.981252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.257447 (+1276195us) service_pool.cc:225] Handling call
0902 01:58:42.257768 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.257916 (+ 148us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.257946 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.257995 (+ 49us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:42.258026 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.258080 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.258169 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.234840 31073 rpcz_store.cc:276] 0902 01:58:40.918225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.197526 (+1279301us) service_pool.cc:225] Handling call
0902 01:58:42.203838 (+ 6312us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.203985 (+ 147us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.204008 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.204049 (+ 41us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:42.204076 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.204118 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.204207 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.200245 30917 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:42.179620 32708 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
W20250902 01:58:42.165534 30892 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.025s sys 0.084s
I20250902 01:58:42.257859 31000 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:58:42.260711 30892 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.783s user 0.025s sys 0.084s
W20250902 01:58:42.201673 30926 rpcz_store.cc:269] 0902 01:58:38.343638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.343745 (+ 107us) service_pool.cc:225] Handling call
0902 01:58:42.196858 (+3853113us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540021d754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:42.196976 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1061533440}
I20250902 01:58:42.171806 388 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.194756 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 395) took 1276 ms. Trace:
I20250902 01:58:42.221920 30964 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:42.232966 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:42.204483 32758 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.212028 32728 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:42.251417 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 164717us
I20250902 01:58:42.263051 31070 rpcz_store.cc:276] 0902 01:58:40.917920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.169419 (+1251499us) service_pool.cc:225] Handling call
0902 01:58:42.194256 (+ 24837us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.194455 (+ 199us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.194481 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.194522 (+ 41us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:42.194549 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.194598 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.194706 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:42.252751 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.003s sys 0.004s
W20250902 01:58:42.234380 30922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.017s sys 0.027s
W20250902 01:58:42.276290 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s user 0.003s sys 0.004s
W20250902 01:58:42.276343 30922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.017s sys 0.027s
W20250902 01:58:42.260974 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 914398us
W20250902 01:58:42.276633 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 850041us
I20250902 01:58:42.263734 30964 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:42.277117 31038 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:42.275995 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:42.219460 396 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.276973 30892 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:42.279217 447 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:42.267987 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.278128 31038 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:42.263949 32672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.008s sys 0.005s
I20250902 01:58:42.284328 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 310) took 1339 ms. Trace:
I20250902 01:58:42.285346 30892 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:42.193825 430 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.289160 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:42.264569 343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.083s user 0.009s sys 0.015s
I20250902 01:58:42.292752 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.228209 425 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:42.286829 32672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s user 0.008s sys 0.005s
I20250902 01:58:42.269109 32754 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:42.295236 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1543) took 2034 ms. Trace:
I20250902 01:58:42.282270 378 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:42.297726 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 398) took 1279 ms. Trace:
I20250902 01:58:42.286852 30922 rpcz_store.cc:276] 0902 01:58:40.944632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.946557 (+ 1925us) service_pool.cc:225] Handling call
0902 01:58:40.984000 (+ 37443us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:40.984162 (+ 162us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:58:41.494597 (+510435us) log.cc:844] Serialized 14 byte log entry
0902 01:58:42.284051 (+789454us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.284101 (+ 50us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:42.284136 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.284189 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.284278 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4170,"prepare.queue_time_us":1140931,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":7998,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"thread_start_us":1264571,"threads_started":2,"wal-append.queue_time_us":669627}
I20250902 01:58:42.259774 433 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.282413 32728 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.286639 348 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.381801 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:42.576572 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:42.753204 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.286254 32758 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:42.320984 430 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.316771 362 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.458s
I20250902 01:58:42.736948 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.773607 30924 raft_consensus.cc:2391] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:42.292800 372 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:41.540807 31723 rpcz_store.cc:276] 0902 01:58:37.640368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.647942 (+ 7574us) service_pool.cc:225] Handling call
0902 01:58:37.725493 (+ 77551us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:37.754031 (+ 28538us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:37.754388 (+ 357us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:37.761368 (+ 6980us) log.cc:844] Serialized 26 byte log entry
0902 01:58:38.276214 (+514846us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:38.588845 (+312631us) log.cc:844] Serialized 14 byte log entry
0902 01:58:38.610488 (+ 21643us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:38.626677 (+ 16189us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:40.514765 (+1888088us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:41.031367 (+516602us) raft_consensus.cc:1683] finished
0902 01:58:41.052289 (+ 20922us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:41.092922 (+ 40633us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47388,"prepare.queue_time_us":453373,"prepare.run_cpu_time_us":6536,"prepare.run_wall_time_us":186777,"raft.queue_time_us":32829,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":29220,"spinlock_wait_cycles":30420864,"thread_start_us":732168,"threads_started":2,"wal-append.queue_time_us":693060}
W20250902 01:58:42.519376 31692 leader_election.cc:336] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.546s (SENT)
I20250902 01:58:42.292587 32568 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:42.753901 343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.633s user 0.018s sys 0.032s
I20250902 01:58:42.296124 349 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:42.784792 31072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1533) took 2849 ms (client timeout 2090 ms). Trace:
I20250902 01:58:42.299846 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:42.753341 31075 rpcz_store.cc:276] 0902 01:58:40.261227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.437967 (+176740us) service_pool.cc:225] Handling call
0902 01:58:40.476119 (+ 38152us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:40.534688 (+ 58569us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:40.534730 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:40.544633 (+ 9903us) log.cc:844] Serialized 26 byte log entry
0902 01:58:40.824526 (+279893us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:40.824580 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.287211 (+1462631us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x7b540036e554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:42.287254 (+ 43us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:42.287299 (+ 45us) raft_consensus.cc:1683] finished
0902 01:58:42.287314 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.295165 (+ 7851us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12843,"raft.queue_time_us":20343,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":10861,"spinlock_wait_cycles":104179968,"thread_start_us":179468,"threads_started":1,"wal-append.queue_time_us":459691,"wal-append.run_cpu_time_us":3854,"wal-append.run_wall_time_us":1675097}
W20250902 01:58:42.566322 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 62263.9us
I20250902 01:58:42.672292 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.404481 446 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:42.603068 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49554 (request call id 0) took 3990 ms. Trace:
I20250902 01:58:42.601065 32750 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:42.599764 32723 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250902 01:58:42.749432 422 log.cc:927] Time spent T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.676s user 0.002s sys 0.000s
I20250902 01:58:42.384603 32720 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:42.817010 32720 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:42.817420 32720 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:42.688736 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.425694 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.634011 356 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.825155 32568 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:42.801797 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:42.320053 32706 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:42.603680 305 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:42.390779 30919 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.380s user 0.011s sys 0.000s
I20250902 01:58:42.297767 426 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:42.806320 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 506576us
I20250902 01:58:42.882697 30919 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:42.610833 302 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:42.808554 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:42.883018 425 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:42.808709 446 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:42.883347 30919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 316) took 1936 ms (client timeout 1610 ms). Trace:
I20250902 01:58:42.883505 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:42.883733 30919 rpcz_store.cc:269] 0902 01:58:40.946430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.006162 (+ 59732us) service_pool.cc:225] Handling call
0902 01:58:42.883267 (+1877105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.598445 347 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:42.884763 347 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.770825 32758 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:42.885643 32758 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:42.291283 447 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:42.591776 303 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:42.775242 372 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:42.811108 32750 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:42.753948 31067 rpcz_store.cc:276] 0902 01:58:41.018299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.236462 (+1218163us) service_pool.cc:225] Handling call
0902 01:58:42.282490 (+ 46028us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.290398 (+ 7908us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.292703 (+ 2305us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.297429 (+ 4726us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:42.297478 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.297555 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:42.297674 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:42.883729 446 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:42.889683 32750 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:42.890579 32750 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:42.814711 32723 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:42.887306 30923 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:42.792320 420 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:42.778259 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 258966us
I20250902 01:58:42.802891 412 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:42.603929 448 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250902 01:58:42.863781 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.000s sys 0.019s
W20250902 01:58:42.868616 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.280s user 0.014s sys 0.006s
I20250902 01:58:42.882150 338 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:42.903589 31694 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.009s sys 0.012s
W20250902 01:58:42.948609 31694 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.009s sys 0.012s
W20250902 01:58:42.948782 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.013s sys 0.053s
W20250902 01:58:42.949139 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.013s sys 0.053s
I20250902 01:58:42.882772 426 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:42.806152 31072 rpcz_store.cc:269] 0902 01:58:39.935387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.950622 (+ 15235us) service_pool.cc:225] Handling call
0902 01:58:42.381616 (+2430994us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:42.784715 (+403099us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3014016}
I20250902 01:58:42.883562 31075 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:42.887349 372 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:42.908154 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.320s user 0.014s sys 0.006s
W20250902 01:58:42.908264 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.000s sys 0.020s
W20250902 01:58:42.908816 30994 consensus_meta.cc:287] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.683s user 0.014s sys 0.008s
I20250902 01:58:42.977288 30994 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:42.977905 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1768) took 1578 ms. Trace:
I20250902 01:58:42.978286 30994 rpcz_store.cc:276] 0902 01:58:41.399478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.180321 (+780843us) service_pool.cc:225] Handling call
0902 01:58:42.977835 (+797514us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:42.891360 432 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.372s user 0.001s sys 0.000s
W20250902 01:58:42.954660 31039 leader_election.cc:336] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.502s (SENT)
W20250902 01:58:42.954493 31724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.000s sys 0.007s
W20250902 01:58:42.954025 30993 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.005s sys 0.004s
W20250902 01:58:42.900810 32711 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.738s user 0.013s sys 0.010s
I20250902 01:58:42.997723 32711 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:42.959933 30892 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:42.984951 338 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.002583 30921 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250902 01:58:43.003715 32711 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.008147 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1492) took 2104 ms. Trace:
I20250902 01:58:43.008550 31071 rpcz_store.cc:276] 0902 01:58:40.904064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.910456 (+ 6392us) service_pool.cc:225] Handling call
0902 01:58:40.910771 (+ 315us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:42.049729 (+1138958us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.007871 (+958142us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400201054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:43.008079 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1183838080}
I20250902 01:58:42.975419 426 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:42.848672 32568 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:42.891086 32723 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:42.883738 305 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:42.974244 324 consensus_meta.cc:287] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.832s user 0.010s sys 0.015s
W20250902 01:58:42.977910 30965 connection.cc:232] RPC call timeout handler was delayed by -1.3437494969999761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20250902 01:58:42.984895 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:42.984843 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:43.056700 324 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.057853 30998 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:58:43.056807 30965 connection.cc:232] RPC call timeout handler was delayed by -1.2798494969999865s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20250902 01:58:43.059008 30965 connection.cc:232] RPC call timeout handler was delayed by -1.1914381139999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
W20250902 01:58:43.059371 30965 connection.cc:232] RPC call timeout handler was delayed by -1.1786170519999359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20250902 01:58:43.059877 30965 connection.cc:232] RPC call timeout handler was delayed by -1.125949496999965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
W20250902 01:58:43.060204 30965 connection.cc:232] RPC call timeout handler was delayed by -1.117849496999952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
I20250902 01:58:42.985280 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:43.061448 31075 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:42.887230 447 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:42.976795 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 697797us
W20250902 01:58:43.009627 310 consensus_meta.cc:287] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.943s user 0.004s sys 0.004s
I20250902 01:58:42.908003 32751 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:42.997211 30993 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.005s sys 0.004s
I20250902 01:58:42.996217 32672 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:42.809155 31722 rpcz_store.cc:276] 0902 01:58:38.641291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.663118 (+ 21827us) service_pool.cc:225] Handling call
0902 01:58:42.602653 (+3939535us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54005909a0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c4c50 00007fc60212a612 00007fc6070901ba 00007fc60072a1c4 00007fc60072a152 00007fc60072a081 00007fc600729ffc 00007fc6007292b2 00007fc60052fccc 00007fc60052f106
0902 01:58:42.602985 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2251415936}
I20250902 01:58:43.033485 32568 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:43.009642 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:40.952983 31069 rpcz_store.cc:276] 0902 01:58:39.886272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.890182 (+ 3910us) service_pool.cc:225] Handling call
0902 01:58:39.947014 (+ 56832us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.947196 (+ 182us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:40.183248 (+236052us) log.cc:844] Serialized 14 byte log entry
0902 01:58:40.950512 (+767264us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:40.950574 (+ 62us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:40.950627 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:40.950706 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:40.950843 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5456,"raft.queue_time_us":15661,"thread_start_us":747336,"threads_started":1}
I20250902 01:58:42.891213 396 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:42.983880 31070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.001s sys 0.009s
W20250902 01:58:43.345603 31070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.951s user 0.001s sys 0.009s
W20250902 01:58:43.138602 30965 leader_election.cc:336] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.613s (SENT)
I20250902 01:58:43.143918 32728 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.144510 324 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:43.153031 32719 consensus_meta.cc:287] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.953s user 0.003s sys 0.005s
W20250902 01:58:43.154641 319 consensus_meta.cc:287] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.941s user 0.007s sys 0.002s
I20250902 01:58:43.011312 433 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:43.167703 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.005s sys 0.028s
W20250902 01:58:43.383795 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.147s user 0.006s sys 0.028s
W20250902 01:58:43.242080 31721 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.009s sys 0.012s
W20250902 01:58:43.294641 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.017s user 0.026s sys 0.067s
W20250902 01:58:43.395040 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.117s user 0.026s sys 0.067s
W20250902 01:58:43.395447 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.13184e+06us
I20250902 01:58:43.346534 30965 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:43.233647 32751 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:43.373379 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 310480us
W20250902 01:58:43.400298 31693 connection.cc:232] RPC call timeout handler was delayed by -1.0767460740000843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20250902 01:58:43.375267 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1549) took 1250 ms. Trace:
I20250902 01:58:43.400489 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:43.375856 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 421276us
I20250902 01:58:43.400753 31070 rpcz_store.cc:276] 0902 01:58:42.124457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.282987 (+158530us) service_pool.cc:225] Handling call
0902 01:58:42.291769 (+ 8782us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:42.291955 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.291977 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:42.311685 (+ 19708us) log.cc:844] Serialized 26 byte log entry
0902 01:58:43.346056 (+1034371us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:43.346108 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.346191 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:43.346258 (+ 67us) raft_consensus.cc:1683] finished
0902 01:58:43.346275 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.375203 (+ 28928us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":951178,"threads_started":1,"wal-append.queue_time_us":616739}
W20250902 01:58:43.378001 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:43.300328 32708 consensus_meta.cc:287] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.040s user 0.006s sys 0.002s
W20250902 01:58:43.376602 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 416813us
W20250902 01:58:43.405339 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:43.376647 30890 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.005s sys 0.012s
W20250902 01:58:43.408156 30890 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.989s user 0.005s sys 0.012s
I20250902 01:58:43.376940 32709 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:43.236802 310 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.413583 30892 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:43.414093 31693 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:43.332844 437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.004s sys 0.018s
I20250902 01:58:43.426111 324 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:43.426424 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:43.429066 32723 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.429512 32568 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:43.432044 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 293507us
I20250902 01:58:43.698954 32568 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:43.433151 32720 consensus_meta.cc:287] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.008s
I20250902 01:58:43.296655 32672 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:43.700979 32672 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:43.434357 332 consensus_meta.cc:287] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.222s user 0.004s sys 0.005s
I20250902 01:58:43.435142 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1505) took 2526 ms. Trace:
I20250902 01:58:43.297281 32758 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:43.701519 332 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.701534 31074 rpcz_store.cc:276] 0902 01:58:40.911802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.038213 (+126411us) service_pool.cc:225] Handling call
0902 01:58:41.038987 (+ 774us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.049216 (+1010229us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.434868 (+1385652us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b540024b3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:43.435076 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1619922816}
I20250902 01:58:43.702695 32758 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.702837 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:43.703306 31074 raft_consensus.cc:2374] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3. Re-sending same reply.
I20250902 01:58:43.707768 31073 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
W20250902 01:58:43.435897 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:43.668020 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 261719us
W20250902 01:58:43.683568 31692 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.012s sys 0.005s
W20250902 01:58:43.646759 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 56767.4us
W20250902 01:58:43.711195 31692 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.733s user 0.013s sys 0.005s
I20250902 01:58:43.659471 30964 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:43.711486 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 733142us
W20250902 01:58:43.677001 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 268100us
I20250902 01:58:43.708439 32738 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:43.712973 32738 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:43.713184 391 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:43.713326 32738 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:43.713488 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:43.384436 314 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250902 01:58:43.394949 31721 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.096s user 0.010s sys 0.012s
W20250902 01:58:43.575438 446 consensus_meta.cc:287] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.687s user 0.003s sys 0.003s
W20250902 01:58:43.673538 31066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.005s sys 0.009s
W20250902 01:58:43.727952 31000 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1774) took 2236 ms (client timeout 1899 ms). Trace:
W20250902 01:58:43.670137 32718 consensus_meta.cc:287] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 2.009s user 0.011s sys 0.000s
I20250902 01:58:43.414911 32751 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:43.681730 386 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250902 01:58:43.577849 378 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.807s user 0.004s sys 0.001s
W20250902 01:58:43.694808 437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s user 0.006s sys 0.018s
I20250902 01:58:43.413975 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:43.435464 32750 consensus_meta.cc:287] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.523s user 0.010s sys 0.000s
W20250902 01:58:43.733938 31000 rpcz_store.cc:269] 0902 01:58:41.491724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.066232 (+574508us) service_pool.cc:225] Handling call
0902 01:58:43.726545 (+1660313us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000d75d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:43.727886 (+ 1341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":438339968}
I20250902 01:58:43.383410 32719 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.713655 391 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:43.737186 30967 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.020s sys 0.052s
W20250902 01:58:43.756814 30967 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.806s user 0.020s sys 0.052s
W20250902 01:58:43.757135 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 806403us
I20250902 01:58:43.570950 305 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.663400 347 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.388792 319 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.743103 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:43.758812 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1770) took 2355 ms. Trace:
I20250902 01:58:43.764278 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:43.763823 31000 rpcz_store.cc:276] 0902 01:58:41.403056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.757928 (+2354872us) service_pool.cc:225] Handling call
0902 01:58:43.758267 (+ 339us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:43.758440 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:43.758473 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:43.758525 (+ 52us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:43.758566 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.758624 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.758754 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:43.765208 30920 raft_consensus.cc:2366] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3.
W20250902 01:58:43.735144 31066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.005s sys 0.012s
I20250902 01:58:43.570525 32724 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:43.725286 359 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:43.717555 32709 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:43.733458 314 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:43.724538 32728 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:43.742909 32763 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.015s sys 0.073s
I20250902 01:58:43.435822 32708 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:43.708478 32720 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.769806 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:43.790992 30920 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:43.791766 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:43.749789 370 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:43.739723 32718 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.747529 32759 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:43.757305 437 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:43.740967 338 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.738021 446 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:43.762832 332 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.789031 32728 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.783901 314 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:43.794173 469 log.cc:927] Time spent T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20250902 01:58:43.795053 310 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.784260 32709 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.797060 391 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.839037 391 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:43.839479 391 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.840113 32709 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.841147 391 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:43.798703 32708 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:43.799429 31071 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:43.806911 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 512) took 1659 ms. Trace:
I20250902 01:58:43.841967 31724 rpcz_store.cc:276] 0902 01:58:42.147597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.148816 (+ 1219us) service_pool.cc:225] Handling call
0902 01:58:42.201040 (+ 52224us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:42.234891 (+ 33851us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.237533 (+ 2642us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:42.293477 (+ 55944us) log.cc:844] Serialized 26 byte log entry
0902 01:58:43.027407 (+733930us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:43.030328 (+ 2921us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.060886 (+ 30558us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:43.806718 (+745832us) raft_consensus.cc:1683] finished
0902 01:58:43.806743 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.806859 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27617,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":563459,"threads_started":1,"wal-append.queue_time_us":1422965}
I20250902 01:58:43.843716 446 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:43.806931 447 consensus_meta.cc:287] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.558s user 0.005s sys 0.001s
I20250902 01:58:43.808732 32723 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.757158 30919 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:43.748334 30921 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 1.
W20250902 01:58:44.740484 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1679) took 1394 ms (client timeout 1679 ms). Trace:
W20250902 01:58:44.740940 30921 rpcz_store.cc:269] 0902 01:58:43.346144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.361290 (+ 15146us) service_pool.cc:225] Handling call
0902 01:58:43.713323 (+352033us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.740282 (+1026959us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b540023f5d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.740411 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79019264}
I20250902 01:58:44.742177 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.743100 30921 raft_consensus.cc:2408] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:43.987584 472 consensus_queue.cc:1046] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:44.744838 472 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:44.745204 472 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:44.745992 472 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.994989 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.005862 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1675) took 1007 ms. Trace:
I20250902 01:58:44.748471 30926 rpcz_store.cc:276] 0902 01:58:42.998815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.999612 (+ 797us) service_pool.cc:225] Handling call
0902 01:58:44.005172 (+1005560us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:44.005367 (+ 195us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.005533 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:44.005561 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:44.005603 (+ 42us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:44.005636 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.005689 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.005799 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":871869696}
I20250902 01:58:44.750020 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.750528 30926 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
W20250902 01:58:44.750981 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1686) took 1319 ms (client timeout 1727 ms). Trace:
I20250902 01:58:44.060045 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1662) took 4127 ms. Trace:
I20250902 01:58:44.751744 31001 rpcz_store.cc:276] 0902 01:58:39.933041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:39.933197 (+ 156us) service_pool.cc:225] Handling call
0902 01:58:39.933510 (+ 313us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:39.987300 (+ 53790us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:40.513201 (+525901us) log.cc:844] Serialized 14 byte log entry
0902 01:58:41.044670 (+531469us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:41.044809 (+ 139us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:41.044963 (+ 154us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:42.598044 (+1553081us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b1800408220. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:58:44.059825 (+1461781us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b540027c254. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:44.059872 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.059991 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":851656,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":2175,"raft.queue_time_us":12265,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"spinlock_wait_cycles":138069632,"thread_start_us":1038790,"threads_started":3,"wal-append.queue_time_us":886788,"wal-append.run_cpu_time_us":6690,"wal-append.run_wall_time_us":1736821}
I20250902 01:58:44.754583 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:44.755179 31001 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:44.062652 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:44.085048 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:43.799408 30889 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:44.756460 30999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 275) took 3774 ms (client timeout 1610 ms). Trace:
W20250902 01:58:44.116488 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:44.756839 30999 rpcz_store.cc:269] 0902 01:58:40.982391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.025474 (+ 43083us) service_pool.cc:225] Handling call
0902 01:58:44.084902 (+3059428us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.756375 (+671473us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6152448}
W20250902 01:58:44.757498 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:44.756729 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 957459us
I20250902 01:58:44.758131 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:44.758759 30999 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:44.126893 30824 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44316 (request call id 366) took 6386 ms. Trace:
W20250902 01:58:44.140729 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:44.759555 30889 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:44.143226 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:44.160833 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:44.164891 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 351104us
I20250902 01:58:44.761656 32583 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:44.763440 315 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:44.176120 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1521) took 2641 ms. Trace:
W20250902 01:58:43.829622 31038 connection.cc:232] RPC call timeout handler was delayed by -1.6502345820000528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20250902 01:58:44.767606 31038 connection.cc:232] RPC call timeout handler was delayed by -1.5260345820000794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20250902 01:58:44.768074 31038 connection.cc:232] RPC call timeout handler was delayed by -1.4693345820001014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
W20250902 01:58:44.768370 31038 connection.cc:232] RPC call timeout handler was delayed by -1.3733726000000388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20250902 01:58:44.768646 31038 connection.cc:232] RPC call timeout handler was delayed by -1.3022726000000375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
W20250902 01:58:44.768900 31038 connection.cc:232] RPC call timeout handler was delayed by -1.2920345820000421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20250902 01:58:44.759580 30824 rpcz_store.cc:276] 0902 01:58:37.740835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.740974 (+ 139us) service_pool.cc:225] Handling call
0902 01:58:44.126818 (+6385844us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:39.354634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:39.479743 (+125109us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:39.479802 (+ 59us) write_op.cc:435] Acquired schema lock
0902 01:58:39.479822 (+ 20us) tablet.cc:660] Decoding operations
0902 01:58:39.522727 (+ 42905us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:39.522783 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:39.522808 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0902 01:58:39.522975 (+ 167us) tablet.cc:699] Row locks acquired
0902 01:58:39.522988 (+ 13us) write_op.cc:260] PREPARE: finished
0902 01:58:39.523122 (+ 134us) write_op.cc:270] Start()
0902 01:58:39.523231 (+ 109us) write_op.cc:276] Timestamp: P: 1756778319523082 usec, L: 0
0902 01:58:39.523246 (+ 15us) op_driver.cc:348] REPLICATION: starting
0902 01:58:39.523818 (+ 572us) log.cc:844] Serialized 541 byte log entry
0902 01:58:41.669742 (+2145924us) op_driver.cc:464] REPLICATION: finished
0902 01:58:42.234011 (+564269us) write_op.cc:301] APPLY: starting
0902 01:58:42.234084 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:42.234186 (+ 102us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:42.280449 (+ 46263us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:42.292822 (+ 12373us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:42.321844 (+ 29022us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0902 01:58:42.321885 (+ 41us) write_op.cc:312] APPLY: finished
0902 01:58:42.415119 (+ 93234us) log.cc:844] Serialized 23 byte log entry
0902 01:58:42.442062 (+ 26943us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:42.570812 (+128750us) write_op.cc:454] Released schema lock
0902 01:58:42.578689 (+ 7877us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:42.790746 (+212057us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:42.801519 (+ 10773us) write_op.cc:454] Released schema lock
0902 01:58:44.058370 (+1256851us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b180030ca60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cb805 00007fc6020c48c0 00007fc6020be277 00007fc6020bd569
Metrics: {"mutex_wait_us":251761,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":482978,"apply.run_cpu_time_us":9043,"apply.run_wall_time_us":524914,"num_ops":1,"prepare.queue_time_us":756586,"prepare.run_cpu_time_us":54139,"prepare.run_wall_time_us":516699,"raft.queue_time_us":961753,"raft.run_cpu_time_us":63112,"raft.run_wall_time_us":718385,"replication_time_us":2026464,"spinlock_wait_cycles":9284864,"thread_start_us":2128321,"threads_started":5,"wal-append.queue_time_us":455386}]]}
I20250902 01:58:43.997974 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:43.845371 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1540) took 3573 ms. Trace:
I20250902 01:58:43.980465 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 400) took 2961 ms. Trace:
I20250902 01:58:44.069322 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1513) took 1083 ms. Trace:
I20250902 01:58:44.800355 31071 rpcz_store.cc:276] 0902 01:58:42.985768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.009378 (+ 23610us) service_pool.cc:225] Handling call
0902 01:58:44.069097 (+1059719us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b54000f41d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.069244 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1156973312}
I20250902 01:58:44.801759 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.802713 31071 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:44.803138 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1520) took 1028 ms. Trace:
I20250902 01:58:44.066334 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 401) took 3035 ms. Trace:
I20250902 01:58:44.069867 31074 raft_consensus.cc:1238] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:44.160414 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.213804 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.215335 30996 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:44.216523 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:44.161648 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.224187 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 269) took 3245 ms. Trace:
W20250902 01:58:44.225836 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.015s sys 0.021s
I20250902 01:58:44.230543 30819 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44308 (request call id 327) took 6411 ms. Trace:
I20250902 01:58:44.241438 30823 catalog_manager.cc:5582] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:58:43.918706 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 196107us
W20250902 01:58:43.979593 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 201154us
I20250902 01:58:43.985616 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 764) took 2269 ms. Trace:
I20250902 01:58:43.858414 32718 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.875977 32720 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.874169 32719 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.837194 32728 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:43.837350 32738 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.870810 32751 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.829386 32759 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.872195 319 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:43.788942 324 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20250902 01:58:43.833125 338 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:44.130489 348 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:43.773790 347 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:43.773247 359 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:43.826545 370 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:43.991931 314 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:43.883316 372 consensus_meta.cc:287] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.887s user 0.008s sys 0.000s
I20250902 01:58:43.997578 424 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:43.842200 426 consensus_meta.cc:287] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.780s user 0.003s sys 0.002s
I20250902 01:58:44.075937 428 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:43.831921 437 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:43.844422 447 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.157577 448 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:43.829581 454 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250902 01:58:43.977170 475 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:44.301627 31000 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.007s sys 0.006s
W20250902 01:58:44.305718 30993 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.004s sys 0.008s
I20250902 01:58:44.488327 482 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:43.883911 31039 leader_election.cc:336] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.532s (SENT)
I20250902 01:58:44.635788 30994 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:44.636260 30998 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 265) took 3659 ms (client timeout 2092 ms). Trace:
W20250902 01:58:44.639691 30919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1678) took 1572 ms (client timeout 1532 ms). Trace:
I20250902 01:58:44.727154 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.756562 389 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:44.764801 386 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:44.764748 30923 rpcz_store.cc:276] 0902 01:58:41.534588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.534701 (+ 113us) service_pool.cc:225] Handling call
0902 01:58:42.887097 (+1352396us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.891579 (+ 4482us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.175940 (+1284361us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400382cd0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.176066 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2989666560}
I20250902 01:58:44.875953 386 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:44.876114 30924 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
W20250902 01:58:44.769162 31038 connection.cc:232] RPC call timeout handler was delayed by -1.2398345820000714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
I20250902 01:58:44.876361 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:44.876677 31038 connection.cc:232] RPC call timeout handler was delayed by -1.2230726000000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
W20250902 01:58:44.877030 31038 connection.cc:232] RPC call timeout handler was delayed by -1.1174345820001008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
W20250902 01:58:44.877149 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1684) took 1446 ms (client timeout 1747 ms). Trace:
W20250902 01:58:44.877543 30923 rpcz_store.cc:269] 0902 01:58:43.430968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.876105 (+1445137us) service_pool.cc:225] Handling call
0902 01:58:44.877092 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:44.878072 30924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1674) took 1879 ms (client timeout 2094 ms). Trace:
I20250902 01:58:44.878368 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:44.878482 30924 rpcz_store.cc:269] 0902 01:58:42.998494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.998653 (+ 159us) service_pool.cc:225] Handling call
0902 01:58:44.160687 (+1162034us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.877928 (+717241us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54000609d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.878022 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1137612288}
W20250902 01:58:44.879125 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1699) took 1445 ms (client timeout 1776 ms). Trace:
I20250902 01:58:44.761174 30889 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:44.807997 30925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1673) took 1809 ms (client timeout 1587 ms). Trace:
W20250902 01:58:44.880038 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 118942us
I20250902 01:58:44.776180 30997 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:44.809600 31069 raft_consensus.cc:2391] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:44.880679 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1692) took 1087 ms. Trace:
I20250902 01:58:44.747287 472 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:44.880836 31069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1511) took 1981 ms (client timeout 2127 ms). Trace:
I20250902 01:58:44.774351 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 273) took 3792 ms. Trace:
I20250902 01:58:44.790882 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1671) took 1793 ms. Trace:
I20250902 01:58:44.793542 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:44.743526 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1682) took 1313 ms (client timeout 1621 ms). Trace:
I20250902 01:58:44.881786 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1691) took 1097 ms. Trace:
W20250902 01:58:44.808804 30918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1672) took 1811 ms (client timeout 2066 ms). Trace:
W20250902 01:58:44.751281 30926 rpcz_store.cc:269] 0902 01:58:43.431450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.749766 (+1318316us) service_pool.cc:225] Handling call
0902 01:58:44.750936 (+ 1170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.876279 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1769) took 3476 ms. Trace:
I20250902 01:58:44.881650 389 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:43.941156 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:44.881295 31069 rpcz_store.cc:269] 0902 01:58:42.899949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.161433 (+1261484us) service_pool.cc:225] Handling call
0902 01:58:44.880791 (+719358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.811169 30823 catalog_manager.cc:5582] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:44.885958 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.886117 386 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:44.855459 347 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.886597 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1518) took 1149 ms. Trace:
I20250902 01:58:44.887068 31069 rpcz_store.cc:276] 0902 01:58:43.737380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.885743 (+1148363us) service_pool.cc:225] Handling call
0902 01:58:44.886550 (+ 807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.886549 30823 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:44.856107 359 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:44.863217 426 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:44.885038 31067 raft_consensus.cc:2408] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:44.889995 31070 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:58:44.875332 30919 rpcz_store.cc:269] 0902 01:58:43.066987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.137981 (+ 70994us) service_pool.cc:225] Handling call
0902 01:58:44.639615 (+1501634us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:44.897595 31070 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1512) took 1997 ms (client timeout 2133 ms). Trace:
I20250902 01:58:44.897759 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:44.897957 31070 rpcz_store.cc:269] 0902 01:58:42.903251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.403930 (+500679us) service_pool.cc:225] Handling call
0902 01:58:44.897431 (+1493501us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000f46d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.897546 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2748778880}
I20250902 01:58:44.898494 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1705) took 1137 ms. Trace:
I20250902 01:58:44.898777 30919 rpcz_store.cc:276] 0902 01:58:43.761382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.897527 (+1136145us) service_pool.cc:225] Handling call
0902 01:58:44.898438 (+ 911us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.858616 370 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:44.783478 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:44.882834 31001 rpcz_store.cc:276] 0902 01:58:43.784750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.793312 (+1008562us) service_pool.cc:225] Handling call
0902 01:58:44.881740 (+ 88428us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:44.882876 30918 rpcz_store.cc:269] 0902 01:58:42.997844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.999624 (+ 1780us) service_pool.cc:225] Handling call
0902 01:58:44.216388 (+1216764us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.808753 (+592365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879723904}
I20250902 01:58:44.872601 475 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:44.810362 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.099s user 0.016s sys 0.022s
W20250902 01:58:44.851611 31075 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1560) took 1904 ms (client timeout 2090 ms). Trace:
W20250902 01:58:44.882040 30921 rpcz_store.cc:269] 0902 01:58:43.430491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.741893 (+1311402us) service_pool.cc:225] Handling call
0902 01:58:44.743475 (+ 1582us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.900876 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1519) took 1144 ms. Trace:
I20250902 01:58:44.860354 314 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.840310 338 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:44.901464 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.880987 30997 rpcz_store.cc:276] 0902 01:58:43.793543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.062367 (+268824us) service_pool.cc:225] Handling call
0902 01:58:44.880625 (+818258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.902321 30921 raft_consensus.cc:2391] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:44.902668 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1707) took 1140 ms. Trace:
I20250902 01:58:44.902967 338 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:44.808410 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 272) took 3826 ms. Trace:
I20250902 01:58:44.834988 32751 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:44.883772 30994 rpcz_store.cc:276] 0902 01:58:41.399748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.985443 (+1585695us) service_pool.cc:225] Handling call
0902 01:58:44.635601 (+1650158us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.875155 (+239554us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.876117 (+ 962us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000b6650. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.876231 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2905637248}
I20250902 01:58:44.880761 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:44.877970 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:44.771797 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.900388 30999 raft_consensus.cc:2408] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250902 01:58:44.872822 30993 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.126s user 0.004s sys 0.008s
I20250902 01:58:44.903126 30921 rpcz_store.cc:276] 0902 01:58:43.762017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.901236 (+1139219us) service_pool.cc:225] Handling call
0902 01:58:44.902628 (+ 1392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.911687 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.810223 30992 rpcz_store.cc:276] 0902 01:58:40.979227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.223007 (+1243780us) service_pool.cc:225] Handling call
0902 01:58:43.726659 (+1503652us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:43.727023 (+ 364us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:43.727061 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:43.727565 (+ 504us) log.cc:844] Serialized 26 byte log entry
0902 01:58:44.119105 (+391540us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:44.119156 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.119223 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:44.119275 (+ 52us) raft_consensus.cc:1683] finished
0902 01:58:44.119291 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.223928 (+104637us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54000d75d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.224118 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12770,"raft.queue_time_us":3617,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":6766,"spinlock_wait_cycles":3321901312,"thread_start_us":384503,"threads_started":1,"wal-append.queue_time_us":244916}
I20250902 01:58:44.912345 30921 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:44.912760 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1680) took 1482 ms. Trace:
I20250902 01:58:44.912853 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:44.913033 30921 rpcz_store.cc:276] 0902 01:58:43.429888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.911475 (+1481587us) service_pool.cc:225] Handling call
0902 01:58:44.912719 (+ 1244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.913479 30992 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:44.914045 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:44.879920 30925 rpcz_store.cc:269] 0902 01:58:42.998157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.998262 (+ 105us) service_pool.cc:225] Handling call
0902 01:58:44.213655 (+1215393us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.807905 (+594250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":875060480}
I20250902 01:58:44.900485 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:44.861840 372 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:44.900800 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.24151e+06us
I20250902 01:58:44.884035 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.900485 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.916321 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1697) took 1121 ms. Trace:
I20250902 01:58:44.902614 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:44.898842 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:44.863988 428 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.881456 30922 rpcz_store.cc:276] 0902 01:58:42.997503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.997624 (+ 121us) service_pool.cc:225] Handling call
0902 01:58:44.790298 (+1792674us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:44.790482 (+ 184us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.790630 (+ 148us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:44.790651 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:44.790685 (+ 34us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:44.790715 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.790759 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.790837 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9982848}
I20250902 01:58:44.924026 30997 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:44.836853 32759 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:44.869544 448 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:44.923874 30999 rpcz_store.cc:276] 0902 01:58:43.795237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.783252 (+988015us) service_pool.cc:225] Handling call
0902 01:58:44.916272 (+133020us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.843642 31068 rpcz_store.cc:276] 0902 01:58:41.018896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.979519 (+2960623us) service_pool.cc:225] Handling call
0902 01:58:43.979909 (+ 390us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:43.980103 (+ 194us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:43.980136 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:43.980186 (+ 50us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:43.980224 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.980286 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.980405 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.928287 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:44.873044 482 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.928807 30999 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:44.928881 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:44.929423 31068 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:44.929971 482 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:44.930403 30996 rpcz_store.cc:276] 0902 01:58:40.981536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.259653 (+1278117us) service_pool.cc:225] Handling call
0902 01:58:44.214941 (+1955288us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:44.215137 (+ 196us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.808161 (+593024us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.808356 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4411776}
I20250902 01:58:44.931315 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:44.900817 31075 rpcz_store.cc:269] 0902 01:58:42.902917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.160142 (+1257225us) service_pool.cc:225] Handling call
0902 01:58:44.807496 (+647354us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.874981 31039 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:44.931824 30996 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:44.932267 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1701) took 1119 ms. Trace:
I20250902 01:58:44.932355 30892 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:44.872846 31000 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.081s user 0.007s sys 0.006s
W20250902 01:58:44.875211 30998 rpcz_store.cc:269] 0902 01:58:40.976359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.487698 (+511339us) service_pool.cc:225] Handling call
0902 01:58:44.635453 (+3147755us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000b6654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.636152 (+ 699us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2492584832}
I20250902 01:58:44.804296 31073 rpcz_store.cc:276] 0902 01:58:41.031954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.259873 (+1227919us) service_pool.cc:225] Handling call
0902 01:58:42.286353 (+ 26480us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:43.715066 (+1428713us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.066113 (+351047us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400157a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:44.066276 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3268992384}
I20250902 01:58:44.933763 32749 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:44.935673 32751 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:44.935853 30918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1695) took 1502 ms (client timeout 1948 ms). Trace:
I20250902 01:58:44.907063 347 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:44.916162 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:44.936198 338 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:44.936717 30994 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:44.936606 338 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:44.937026 30926 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:44.937335 338 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:44.937482 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1691) took 1505 ms (client timeout 1831 ms). Trace:
I20250902 01:58:44.833379 32738 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:44.937855 30926 rpcz_store.cc:269] 0902 01:58:43.432475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.883811 (+1451336us) service_pool.cc:225] Handling call
0902 01:58:44.937434 (+ 53623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.938714 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.939271 30926 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:44.939894 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1685) took 1508 ms. Trace:
I20250902 01:58:44.945875 30926 rpcz_store.cc:276] 0902 01:58:43.431201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.938505 (+1507304us) service_pool.cc:225] Handling call
0902 01:58:44.939838 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.824780 31721 rpcz_store.cc:276] 0902 01:58:41.717387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:41.742166 (+ 24779us) service_pool.cc:225] Handling call
0902 01:58:42.087283 (+345117us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.128425 (+ 41142us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:42.289356 (+160931us) log.cc:844] Serialized 14 byte log entry
0902 01:58:43.979316 (+1689960us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:43.981746 (+ 2430us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:43.982486 (+ 740us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.983981 (+ 1495us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.985158 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":486377,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":100545,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":1758200,"threads_started":3,"wal-append.queue_time_us":1517132,"wal-append.run_cpu_time_us":13480,"wal-append.run_wall_time_us":1001898}
W20250902 01:58:44.935945 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 58054us
I20250902 01:58:44.952595 386 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.950889 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.953207 31038 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:44.881426 30995 rpcz_store.cc:276] 0902 01:58:40.981835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.384394 (+1402559us) service_pool.cc:225] Handling call
0902 01:58:42.589238 (+204844us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.806085 (+216847us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:42.810289 (+ 4204us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:42.813803 (+ 3514us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:42.815929 (+ 2126us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.774081 (+1958152us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54002be154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:44.774154 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.774287 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":396075776}
I20250902 01:58:44.925567 475 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.866406 437 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.954412 370 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:44.954952 30995 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250902 01:58:44.935932 428 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.954945 370 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:44.955631 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1775) took 3408 ms. Trace:
I20250902 01:58:44.955765 370 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:44.955979 30995 rpcz_store.cc:276] 0902 01:58:41.581310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.954455 (+3373145us) service_pool.cc:225] Handling call
0902 01:58:44.954738 (+ 283us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:44.955399 (+ 661us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.955581 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.957643 30995 raft_consensus.cc:2953] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:58:44.914170 30917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1676) took 1915 ms (client timeout 2189 ms). Trace:
I20250902 01:58:44.944891 372 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:44.810464 30819 rpcz_store.cc:276] 0902 01:58:37.818832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:37.818969 (+ 137us) service_pool.cc:225] Handling call
0902 01:58:44.230469 (+6411500us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:39.947476 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:40.051881 (+104405us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:40.055357 (+ 3476us) write_op.cc:435] Acquired schema lock
0902 01:58:40.061160 (+ 5803us) tablet.cc:660] Decoding operations
0902 01:58:40.080831 (+ 19671us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:40.082917 (+ 2086us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:40.087961 (+ 5044us) tablet.cc:683] Acquiring locks for 1 operations
0902 01:58:40.100914 (+ 12953us) tablet.cc:699] Row locks acquired
0902 01:58:40.102166 (+ 1252us) write_op.cc:260] PREPARE: finished
0902 01:58:40.109854 (+ 7688us) write_op.cc:270] Start()
0902 01:58:40.112341 (+ 2487us) write_op.cc:276] Timestamp: P: 1756778320109809 usec, L: 0
0902 01:58:40.114354 (+ 2013us) op_driver.cc:348] REPLICATION: starting
0902 01:58:40.170416 (+ 56062us) log.cc:844] Serialized 536 byte log entry
0902 01:58:44.060946 (+3890530us) op_driver.cc:464] REPLICATION: finished
0902 01:58:44.096115 (+ 35169us) write_op.cc:301] APPLY: starting
0902 01:58:44.096192 (+ 77us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:44.096275 (+ 83us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:44.096391 (+ 116us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:44.096615 (+ 224us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:44.096764 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0902 01:58:44.096809 (+ 45us) write_op.cc:312] APPLY: finished
0902 01:58:44.096990 (+ 181us) log.cc:844] Serialized 23 byte log entry
0902 01:58:44.097185 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:44.097271 (+ 86us) write_op.cc:454] Released schema lock
0902 01:58:44.097318 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:44.097930 (+ 612us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:44.097958 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":201827,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":34932,"apply.run_cpu_time_us":1537,"apply.run_wall_time_us":1837,"num_ops":1,"prepare.queue_time_us":1283532,"prepare.run_cpu_time_us":3690,"prepare.run_wall_time_us":242421,"replication_time_us":3946450,"thread_start_us":34613,"threads_started":1}]]}
I20250902 01:58:44.796880 31066 rpcz_store.cc:276] 0902 01:58:40.259120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:40.263604 (+ 4484us) service_pool.cc:225] Handling call
0902 01:58:42.725874 (+2462270us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:42.726114 (+ 240us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:42.726369 (+ 255us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:42.955218 (+228849us) log.cc:844] Serialized 14 byte log entry
0902 01:58:43.779854 (+824636us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:43.785186 (+ 5332us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:43.789554 (+ 4368us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:43.823552 (+ 33998us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:43.832149 (+ 8597us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":306383,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":2056,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":6205,"spinlock_wait_cycles":5014912,"thread_start_us":1040736,"threads_started":2,"wal-append.queue_time_us":823829}
I20250902 01:58:44.806690 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1550) took 2484 ms. Trace:
I20250902 01:58:44.963590 31074 rpcz_store.cc:276] 0902 01:58:42.322253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.712389 (+1390136us) service_pool.cc:225] Handling call
0902 01:58:44.069698 (+357309us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.805614 (+735916us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.806521 (+ 907us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54000f41d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:44.806641 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":816804608}
I20250902 01:58:44.964155 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.964599 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.964728 31066 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:44.917804 389 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:44.965622 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.965516 370 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.946161 426 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:44.879478 30923 rpcz_store.cc:269] 0902 01:58:43.434133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.878107 (+1443974us) service_pool.cc:225] Handling call
0902 01:58:44.879077 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.966385 31066 raft_consensus.cc:2408] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:58:44.966765 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:44.967032 389 consensus_queue.cc:1046] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:44.868744 447 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:44.947829 31070 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:44.967133 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:44.947427 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1551) took 2562 ms. Trace:
I20250902 01:58:44.953760 30919 raft_consensus.cc:2408] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250902 01:58:44.922683 30967 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.599s (SENT) [suppressed 61 similar messages]
W20250902 01:58:45.012879 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 90234.1us
I20250902 01:58:44.979445 448 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.918226 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:44.901475 31067 rpcz_store.cc:276] 0902 01:58:43.756735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.726832 (+970097us) service_pool.cc:225] Handling call
0902 01:58:44.900685 (+173853us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.959118 30996 rpcz_store.cc:276] 0902 01:58:43.812868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.931126 (+1118258us) service_pool.cc:225] Handling call
0902 01:58:44.932216 (+ 1090us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:44.932446 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.04864e+06us
W20250902 01:58:45.020310 31039 connection.cc:232] RPC call timeout handler was delayed by -1.1159297619999506s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32930000000000004s for a response
W20250902 01:58:45.020897 31039 leader_election.cc:336] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.587s (SENT)
I20250902 01:58:45.021322 31039 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:45.022318 31039 connection.cc:232] RPC call timeout handler was delayed by -1.025677117999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20250902 01:58:45.022694 31039 connection.cc:232] RPC call timeout handler was delayed by -1.009298418999947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1948s for a response
I20250902 01:58:44.961639 31001 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:44.919154 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.936642 30964 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:44.962162 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:45.024291 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 87717us
W20250902 01:58:45.024575 30924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1683) took 1593 ms (client timeout 1828 ms). Trace:
W20250902 01:58:45.025025 30924 rpcz_store.cc:269] 0902 01:58:43.430734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.879355 (+1448621us) service_pool.cc:225] Handling call
0902 01:58:45.024513 (+145158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.025403 30964 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:45.025916 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.924372 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.896580 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:44.937180 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1698) took 1126 ms. Trace:
W20250902 01:58:44.962170 30917 rpcz_store.cc:269] 0902 01:58:42.999146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.999782 (+ 636us) service_pool.cc:225] Handling call
0902 01:58:44.771666 (+1771884us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:44.914118 (+142452us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3134720}
I20250902 01:58:45.028321 31072 raft_consensus.cc:2391] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:58:44.936290 30918 rpcz_store.cc:269] 0902 01:58:43.433296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.900298 (+1467002us) service_pool.cc:225] Handling call
0902 01:58:44.935807 (+ 35509us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.026714 30922 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:45.029060 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1709) took 1266 ms. Trace:
I20250902 01:58:45.029399 30922 rpcz_store.cc:276] 0902 01:58:43.762574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.924142 (+1161568us) service_pool.cc:225] Handling call
0902 01:58:45.028994 (+104852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.030190 30994 rpcz_store.cc:276] 0902 01:58:43.811122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.915951 (+1104829us) service_pool.cc:225] Handling call
0902 01:58:44.937127 (+ 21176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.976141 389 raft_consensus.cc:3053] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:44.918300 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1704) took 1178 ms. Trace:
I20250902 01:58:44.976622 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:44.973290 359 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:44.951239 30999 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:45.033625 30926 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:45.033367 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1777) took 2968 ms. Trace:
I20250902 01:58:45.036672 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1702) took 1223 ms. Trace:
W20250902 01:58:45.036782 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1681) took 1606 ms (client timeout 2075 ms). Trace:
I20250902 01:58:45.037042 30999 rpcz_store.cc:276] 0902 01:58:43.813475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.928049 (+1114574us) service_pool.cc:225] Handling call
0902 01:58:45.036625 (+108576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.979466 31070 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:45.036932 30996 rpcz_store.cc:276] 0902 01:58:42.065111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.032739 (+2967628us) service_pool.cc:225] Handling call
0902 01:58:45.032963 (+ 224us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.033107 (+ 144us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.033128 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.033163 (+ 35us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:45.033191 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.033233 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.033319 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:45.037142 30926 rpcz_store.cc:269] 0902 01:58:43.430204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.976414 (+1546210us) service_pool.cc:225] Handling call
0902 01:58:45.036736 (+ 60322us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.038726 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.039134 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.039676 31070 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.037318 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.040894 30892 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:44.965626 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:45.036537 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1708) took 1274 ms. Trace:
I20250902 01:58:45.042111 30924 rpcz_store.cc:276] 0902 01:58:43.762290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.025673 (+1263383us) service_pool.cc:225] Handling call
0902 01:58:45.036481 (+ 10808us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.042870 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.043131 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1702) took 1608 ms. Trace:
I20250902 01:58:44.984920 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.043421 30918 rpcz_store.cc:276] 0902 01:58:43.434982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.032720 (+1597738us) service_pool.cc:225] Handling call
0902 01:58:45.043088 (+ 10368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.998690 31068 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:45.044098 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1562) took 1307 ms. Trace:
I20250902 01:58:45.012439 31075 rpcz_store.cc:276] 0902 01:58:42.384891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.933424 (+2548533us) service_pool.cc:225] Handling call
0902 01:58:44.947008 (+ 13584us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.947158 (+ 150us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:44.947179 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:44.947216 (+ 37us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:44.947246 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:44.947293 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:44.947378 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.944227 30920 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:44.803519 31071 rpcz_store.cc:276] 0902 01:58:43.774780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.801419 (+1026639us) service_pool.cc:225] Handling call
0902 01:58:44.803075 (+ 1656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.967232 370 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:45.016640 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 98488.6us
I20250902 01:58:45.023923 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1706) took 1262 ms. Trace:
I20250902 01:58:44.967447 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1700) took 1533 ms. Trace:
I20250902 01:58:45.050666 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.027382 31069 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:44.980360 30997 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:45.051087 30923 rpcz_store.cc:276] 0902 01:58:43.434447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.966548 (+1532101us) service_pool.cc:225] Handling call
0902 01:58:44.967397 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.017266 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1686) took 3548 ms. Trace:
I20250902 01:58:45.048928 31074 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.051810 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1696) took 1256 ms. Trace:
I20250902 01:58:45.052035 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.052115 30997 rpcz_store.cc:276] 0902 01:58:43.794949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.902413 (+1107464us) service_pool.cc:225] Handling call
0902 01:58:45.051763 (+149350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.052520 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.052929 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:45.053141 31074 raft_consensus.cc:2391] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.053309 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1694) took 1258 ms. Trace:
I20250902 01:58:45.053936 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.054817 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.055029 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.055805 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:45.055915 359 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:45.057116 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:45.057665 31071 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:45.050159 428 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:45.060034 428 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:45.061923 428 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:45.050753 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:45.053809 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.050184 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.050571 30925 rpcz_store.cc:276] 0902 01:58:43.761700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.918772 (+1157072us) service_pool.cc:225] Handling call
0902 01:58:45.023871 (+105099us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.049400 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:45.044395 31068 rpcz_store.cc:276] 0902 01:58:43.736315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.928632 (+1192317us) service_pool.cc:225] Handling call
0902 01:58:45.044051 (+115419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:44.953161 338 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:45.033962 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.033366 372 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:45.354460 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:45.126170 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 348467us
I20250902 01:58:45.705660 30924 raft_consensus.cc:2374] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3. Re-sending same reply.
W20250902 01:58:45.526640 31001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1690) took 1746 ms (client timeout 1809 ms). Trace:
I20250902 01:58:45.124739 30994 raft_consensus.cc:1238] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:45.116714 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.129467 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.707026 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.063629 30918 raft_consensus.cc:2391] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:45.381021 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:45.691920 30998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.000s sys 0.007s
I20250902 01:58:45.028445 426 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:45.038169 447 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:45.721971 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1552) took 3314 ms. Trace:
I20250902 01:58:45.264251 32750 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:45.038420 347 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250902 01:58:45.053607 30997 rpcz_store.cc:276] 0902 01:58:43.794441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.052693 (+1258252us) service_pool.cc:225] Handling call
0902 01:58:45.052905 (+ 212us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.053059 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.053080 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.053114 (+ 34us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:45.053143 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.053188 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.053267 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.033666 482 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:45.752880 30892 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:45.088626 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:45.726302 31068 raft_consensus.cc:2391] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.720791 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:45.756119 31001 rpcz_store.cc:269] 0902 01:58:43.780000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.900287 (+1120287us) service_pool.cc:225] Handling call
0902 01:58:45.117079 (+216792us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b54002bdc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:45.526586 (+409507us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40173824}
I20250902 01:58:45.766129 30925 raft_consensus.cc:2391] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:45.766356 31068 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:45.766870 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1553) took 3201 ms. Trace:
I20250902 01:58:45.767081 31068 rpcz_store.cc:276] 0902 01:58:42.565591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.765927 (+3200336us) service_pool.cc:225] Handling call
0902 01:58:45.766197 (+ 270us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.766710 (+ 513us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.766831 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.768033 31068 raft_consensus.cc:1238] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:45.038309 30919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1703) took 1352 ms (client timeout 1621 ms). Trace:
W20250902 01:58:45.423291 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:45.757002 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1778) took 3669 ms. Trace:
I20250902 01:58:45.773345 31075 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:45.752033 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1688) took 4270 ms. Trace:
I20250902 01:58:45.079546 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:45.283171 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:45.118548 31072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1514) took 2090 ms (client timeout 1754 ms). Trace:
I20250902 01:58:45.774825 30994 rpcz_store.cc:276] 0902 01:58:42.087452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.034227 (+2946775us) service_pool.cc:225] Handling call
0902 01:58:45.124533 (+ 90306us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.756764 (+632231us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.756936 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:45.758363 30918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1688) took 2326 ms (client timeout 3026 ms). Trace:
W20250902 01:58:45.112056 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 61376.3us
I20250902 01:58:45.703474 30917 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:45.460978 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1776) took 3796 ms. Trace:
I20250902 01:58:45.775031 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1558) took 2958 ms. Trace:
I20250902 01:58:45.776557 31068 rpcz_store.cc:276] 0902 01:58:42.817139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.767619 (+2950480us) service_pool.cc:225] Handling call
0902 01:58:45.767874 (+ 255us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.768417 (+ 543us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.774988 (+ 6571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.778724 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:45.779222 30917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1701) took 2344 ms (client timeout 2094 ms). Trace:
W20250902 01:58:45.779446 30917 rpcz_store.cc:269] 0902 01:58:43.434751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.028123 (+1593372us) service_pool.cc:225] Handling call
0902 01:58:45.779176 (+751053us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:45.779887 30922 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1710) took 1993 ms (client timeout 1747 ms). Trace:
W20250902 01:58:45.780098 30918 rpcz_store.cc:269] 0902 01:58:43.431910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.044042 (+1612132us) service_pool.cc:225] Handling call
0902 01:58:45.758305 (+714263us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:45.780237 30922 rpcz_store.cc:269] 0902 01:58:43.786170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.030053 (+1243883us) service_pool.cc:225] Handling call
0902 01:58:45.779828 (+749775us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:45.776430 31072 rpcz_store.cc:269] 0902 01:58:43.027836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.997659 (+969823us) service_pool.cc:225] Handling call
0902 01:58:45.118361 (+1120702us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400020654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:45.118481 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":538080640}
I20250902 01:58:45.117978 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:45.758750 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1711) took 1961 ms (client timeout 1828 ms). Trace:
I20250902 01:58:45.759673 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:45.759771 31693 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:45.049188 386 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:45.782496 30924 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:45.783191 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.781826 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1561) took 2826 ms. Trace:
I20250902 01:58:45.759380 31073 rpcz_store.cc:276] 0902 01:58:42.407866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.934500 (+2526634us) service_pool.cc:225] Handling call
0902 01:58:44.960225 (+ 25725us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:44.960398 (+ 173us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:44.960669 (+ 271us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:44.960692 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:44.961050 (+ 358us) log.cc:844] Serialized 26 byte log entry
0902 01:58:45.131325 (+170275us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:45.131364 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.131421 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:45.712868 (+581447us) raft_consensus.cc:1683] finished
0902 01:58:45.712893 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.721904 (+ 9011us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1031,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"spinlock_wait_cycles":29686528,"thread_start_us":170033,"threads_started":1,"wal-append.queue_time_us":750020}
W20250902 01:58:45.703914 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 384741us
I20250902 01:58:45.707460 31000 rpcz_store.cc:276] 0902 01:58:41.469249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.766242 (+2296993us) service_pool.cc:225] Handling call
0902 01:58:43.768279 (+ 2037us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:43.768511 (+ 232us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:44.933099 (+1164588us) log.cc:844] Serialized 14 byte log entry
0902 01:58:45.015369 (+ 82270us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.015431 (+ 62us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.015496 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.017030 (+ 1534us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.017192 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":204174,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":93,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":16959,"thread_start_us":1223124,"threads_started":2,"wal-append.queue_time_us":116054}
I20250902 01:58:45.762996 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1689) took 1983 ms. Trace:
W20250902 01:58:45.451396 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 428098us
I20250902 01:58:45.787045 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.723292 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1712) took 1925 ms. Trace:
W20250902 01:58:45.117908 30920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1698) took 1684 ms (client timeout 1587 ms). Trace:
I20250902 01:58:45.768525 31074 raft_consensus.cc:2391] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.695183 31724 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:45.771559 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.771878 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1687) took 2340 ms. Trace:
I20250902 01:58:45.124747 32759 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:45.720970 315 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:45.811097 31069 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:45.799858 31001 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:48184
W20250902 01:58:45.517083 29788 debug-util.cc:398] Leaking SignalData structure 0x7b080000fdc0 after lost signal to thread 31039
I20250902 01:58:45.781066 30995 rpcz_store.cc:276] 0902 01:58:41.669256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.957137 (+3287881us) service_pool.cc:225] Handling call
0902 01:58:44.957385 (+ 248us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.957551 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:45.072805 (+115254us) log.cc:844] Serialized 14 byte log entry
0902 01:58:45.460535 (+387730us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.460599 (+ 64us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.460679 (+ 80us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.460764 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.460902 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2884,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":23041,"thread_start_us":387364,"threads_started":1,"wal-append.queue_time_us":646491}
I20250902 01:58:45.767660 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.784667 31072 rpcz_store.cc:276] 0902 01:58:42.955428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.781378 (+2825950us) service_pool.cc:225] Handling call
0902 01:58:45.781548 (+ 170us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.781647 (+ 99us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.781660 (+ 13us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.781684 (+ 24us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:45.781701 (+ 17us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.781728 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.781780 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.789443 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.036700 389 raft_consensus.cc:738] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:45.899151 31725 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:45.897989 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.900053 31072 raft_consensus.cc:2391] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.900501 30924 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:45.900704 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:45.788328 30920 rpcz_store.cc:269] 0902 01:58:43.433930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:43.994741 (+560811us) service_pool.cc:225] Handling call
0902 01:58:45.117731 (+1122990us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54000c21d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:45.117847 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":450948352}
I20250902 01:58:45.316468 489 raft_consensus.cc:3053] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:45.287990 31022 debug-util.cc:398] Leaking SignalData structure 0x7b080016fee0 after lost signal to thread 29783
I20250902 01:58:45.112224 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.789875 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.789763 30925 rpcz_store.cc:276] 0902 01:58:43.431696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.706247 (+2274551us) service_pool.cc:225] Handling call
0902 01:58:45.771840 (+ 65593us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.765151 32750 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:45.901597 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:45.905273 31725 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:45.907956 31722 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:45.908433 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.908558 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.908932 32750 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:45.909082 31072 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.793614 32759 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:45.909516 32759 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:45.909638 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.910444 30920 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:45.910763 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.910979 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1689) took 2478 ms. Trace:
I20250902 01:58:45.911319 31725 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:45.911340 30920 rpcz_store.cc:276] 0902 01:58:43.432097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.910022 (+2477925us) service_pool.cc:225] Handling call
0902 01:58:45.910299 (+ 277us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.910775 (+ 476us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.910928 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.913096 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1693) took 2480 ms. Trace:
I20250902 01:58:45.913380 30920 rpcz_store.cc:276] 0902 01:58:43.432848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.912442 (+2479594us) service_pool.cc:225] Handling call
0902 01:58:45.912671 (+ 229us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.912815 (+ 144us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.912836 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.912875 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:45.912914 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.912960 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.913054 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.914100 31722 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:45.914570 30920 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:45.914955 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.791206 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.659910 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 268) took 4676 ms. Trace:
I20250902 01:58:45.901517 31693 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:45.906459 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.916639 31066 raft_consensus.cc:2408] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:45.654974 448 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:45.405439 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.792395 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:45.706262 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.775197 30998 rpcz_store.cc:276] 0902 01:58:41.481436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.934246 (+3452810us) service_pool.cc:225] Handling call
0902 01:58:44.934480 (+ 234us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:44.934636 (+ 156us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:45.185967 (+251331us) log.cc:844] Serialized 14 byte log entry
0902 01:58:45.751739 (+565772us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.751783 (+ 44us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.751830 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.751888 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.751986 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":340195,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":81,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":816199,"threads_started":2}
W20250902 01:58:45.918378 31022 debug-util.cc:398] Leaking SignalData structure 0x7b080016ff00 after lost signal to thread 29786
I20250902 01:58:45.920428 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.921252 31066 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.037842 454 consensus_queue.cc:1046] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.731s
W20250902 01:58:45.782305 30926 rpcz_store.cc:269] 0902 01:58:43.797176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.038555 (+1241379us) service_pool.cc:225] Handling call
0902 01:58:45.758703 (+720148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.921438 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.922024 454 raft_consensus.cc:3053] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:45.922344 454 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:45.922384 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.923101 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.923206 454 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:45.811103 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:45.907454 489 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:45.924237 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.924860 489 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:45.925248 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.925565 31066 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250902 01:58:45.925731 489 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:45.926236 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 133909us
I20250902 01:58:45.926903 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
W20250902 01:58:45.921125 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08001a4180 after lost signal to thread 30858
W20250902 01:58:45.928359 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08001a0c80 after lost signal to thread 30872
I20250902 01:58:45.928313 30889 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:45.920948 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.923427 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.929857 31070 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:45.916730 344 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:45.920046 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 321) took 1100 ms. Trace:
I20250902 01:58:45.383455 30992 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:45.826495 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.814467 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1699) took 2003 ms. Trace:
I20250902 01:58:45.813069 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1517) took 2116 ms. Trace:
I20250902 01:58:45.931820 30997 rpcz_store.cc:276] 0902 01:58:43.811595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.772269 (+1960674us) service_pool.cc:225] Handling call
0902 01:58:45.813725 (+ 41456us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.814029 (+ 304us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.814077 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.814145 (+ 68us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:45.814213 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.814291 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.814418 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.772935 314 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:45.767987 347 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:45.812422 31001 raft_consensus.cc:604] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:45.786559 30996 rpcz_store.cc:276] 0902 01:58:43.779716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.038514 (+1258798us) service_pool.cc:225] Handling call
0902 01:58:45.038761 (+ 247us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.038923 (+ 162us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:45.317542 (+278619us) log.cc:844] Serialized 14 byte log entry
0902 01:58:45.762552 (+445010us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b1800069580. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d742da
0902 01:58:45.762725 (+ 173us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.762767 (+ 42us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:45.762807 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.762860 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.762953 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":263827,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":36,"raft.queue_time_us":1922,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":15160,"spinlock_wait_cycles":8427776,"thread_start_us":466585,"threads_started":2}
I20250902 01:58:45.890026 32749 consensus_queue.cc:1046] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:45.933466 344 raft_consensus.cc:695] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:58:45.773310 30919 rpcz_store.cc:269] 0902 01:58:43.681322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.899462 (+1218140us) service_pool.cc:225] Handling call
0902 01:58:45.033075 (+133613us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.793681 368 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:45.825508 315 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:45.812885 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.768747 447 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:45.894984 338 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:45.896018 31075 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:45.897158 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.770742 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1783) took 3565 ms. Trace:
I20250902 01:58:45.784107 32756 consensus_queue.cc:1046] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20250902 01:58:45.847834 386 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:45.934695 31722 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:45.787905 30923 rpcz_store.cc:276] 0902 01:58:43.797520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.053633 (+1256113us) service_pool.cc:225] Handling call
0902 01:58:45.723221 (+669588us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.915105 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1694) took 2482 ms. Trace:
I20250902 01:58:45.794541 437 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
W20250902 01:58:45.980517 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 152658us
I20250902 01:58:45.980845 30920 rpcz_store.cc:276] 0902 01:58:43.433060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.914186 (+2481126us) service_pool.cc:225] Handling call
0902 01:58:45.914439 (+ 253us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.914893 (+ 454us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.915061 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.917403 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:45.982295 31072 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:45.917760 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1696) took 2484 ms. Trace:
I20250902 01:58:45.271306 32720 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:45.895743 372 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:45.918583 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 335) took 1069 ms. Trace:
I20250902 01:58:45.985774 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250902 01:58:45.985613 30918 rpcz_store.cc:276] 0902 01:58:44.849324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.784084 (+934760us) service_pool.cc:225] Handling call
0902 01:58:45.918546 (+134462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.920928 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.922448 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 322) took 1102 ms. Trace:
I20250902 01:58:45.987088 30926 rpcz_store.cc:276] 0902 01:58:44.820101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.921819 (+1101718us) service_pool.cc:225] Handling call
0902 01:58:45.922062 (+ 243us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.922190 (+ 128us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.922212 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.922248 (+ 36us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.922278 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.922318 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.922402 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.924345 454 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:45.370769 31071 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:45.925889 31066 pending_rounds.cc:85] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 2
I20250902 01:58:45.818879 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:45.825774 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:45.841646 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.930431 30924 rpcz_store.cc:276] 0902 01:58:44.819834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.919233 (+1099399us) service_pool.cc:225] Handling call
0902 01:58:45.919695 (+ 462us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.919849 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.919865 (+ 16us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.919888 (+ 23us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.919914 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.919944 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.920002 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.931882 31069 rpcz_store.cc:276] 0902 01:58:43.696899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.812361 (+2115462us) service_pool.cc:225] Handling call
0902 01:58:45.812577 (+ 216us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.812744 (+ 167us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.812777 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.812833 (+ 56us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:45.812864 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.812912 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.813015 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.036373 30921 rpcz_store.cc:276] 0902 01:58:43.740206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.913806 (+1173600us) service_pool.cc:225] Handling call
0902 01:58:44.918252 (+ 4446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.898242 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 765) took 1077 ms. Trace:
I20250902 01:58:45.890249 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:45.935961 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.018231 31729 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:45.928692 32759 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:45.935995 30992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1700) took 2122 ms (client timeout 1983 ms). Trace:
I20250902 01:58:45.938339 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.938936 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:45.925762 31727 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:45.933665 32750 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:45.944069 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:45.915568 30993 rpcz_store.cc:276] 0902 01:58:40.978612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:42.216671 (+1238059us) service_pool.cc:225] Handling call
0902 01:58:42.262882 (+ 46211us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:43.345806 (+1082924us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:43.361928 (+ 16122us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:43.655981 (+294053us) log.cc:844] Serialized 26 byte log entry
0902 01:58:44.915736 (+1259755us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:45.034580 (+118844us) log.cc:844] Serialized 14 byte log entry
0902 01:58:45.532610 (+498030us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.542483 (+ 9873us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:45.556643 (+ 14160us) raft_consensus.cc:1683] finished
0902 01:58:45.556663 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.655168 (+ 98505us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":208645,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":73824,"raft.queue_time_us":329520,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8496,"thread_start_us":2672752,"threads_started":4,"wal-append.queue_time_us":944360,"wal-append.run_cpu_time_us":1941,"wal-append.run_wall_time_us":1010234}
W20250902 01:58:45.948789 30965 leader_election.cc:336] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.625s (SENT)
I20250902 01:58:45.970191 30917 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:45.938911 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08003bfc40 after lost signal to thread 30950
I20250902 01:58:45.969831 30999 rpcz_store.cc:276] 0902 01:58:42.205499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.049289 (+2843790us) service_pool.cc:225] Handling call
0902 01:58:45.049534 (+ 245us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.763936 (+714402us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.763965 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.764005 (+ 40us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:45.764037 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.770531 (+ 6494us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54000b6b54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:45.770571 (+ 40us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.770685 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1642659584}
I20250902 01:58:45.936393 347 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:45.937374 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:45.979707 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 196513us
I20250902 01:58:45.899596 389 consensus_queue.cc:260] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:45.983009 30925 rpcz_store.cc:276] 0902 01:58:43.433537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.917024 (+2483487us) service_pool.cc:225] Handling call
0902 01:58:45.917292 (+ 268us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.917466 (+ 174us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.917490 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.917531 (+ 41us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:45.917563 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.917617 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.917711 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:45.983738 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1569) took 1038 ms. Trace:
I20250902 01:58:45.984463 31074 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:45.981561 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 324) took 1135 ms. Trace:
W20250902 01:58:45.989811 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 61566us
I20250902 01:58:45.988001 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:58:45.950902 359 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:46.015684 31073 raft_consensus.cc:1238] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:46.012880 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.016319 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.017166 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 771) took 1024 ms. Trace:
I20250902 01:58:46.014127 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 329) took 1166 ms. Trace:
I20250902 01:58:46.005745 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:45.983804 368 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:46.122316 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:46.131408 31039 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.621s (SENT)
I20250902 01:58:46.127934 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 336) took 1278 ms. Trace:
I20250902 01:58:46.132920 30889 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:46.133034 31039 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:46.134259 31039 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.679s (SENT)
I20250902 01:58:46.132105 30922 rpcz_store.cc:276] 0902 01:58:44.849617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.072390 (+1222773us) service_pool.cc:225] Handling call
0902 01:58:46.127451 (+ 55061us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.127636 (+ 185us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.127662 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.127701 (+ 39us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.127732 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.127787 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.127879 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.134665 30892 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:46.134804 31039 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.135495 30965 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:46.135803 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 130161us
I20250902 01:58:46.132114 30994 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:46.097565 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.138017 30996 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
W20250902 01:58:46.136404 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 187670us
I20250902 01:58:46.135439 30923 rpcz_store.cc:276] 0902 01:58:44.845859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.980849 (+1134990us) service_pool.cc:225] Handling call
0902 01:58:45.981111 (+ 262us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.981295 (+ 184us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:45.981318 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:45.981354 (+ 36us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:45.981384 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.981429 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:45.981517 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:46.138767 30965 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.659s (SENT)
I20250902 01:58:46.139142 30965 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.131143 30926 rpcz_store.cc:276] 0902 01:58:44.847381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.988256 (+1140875us) service_pool.cc:225] Handling call
0902 01:58:46.013566 (+ 25310us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.013742 (+ 176us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.013793 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.013863 (+ 70us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:46.013912 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.013974 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.014082 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.131804 31075 rpcz_store.cc:276] 0902 01:58:44.945714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.055595 (+109881us) service_pool.cc:225] Handling call
0902 01:58:45.983582 (+927987us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540036ef54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:45.983685 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1649303424}
I20250902 01:58:46.141222 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:45.975266 31723 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:46.044634 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:46.018666 31066 pending_rounds.cc:107] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250902 01:58:46.134507 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:46.006850 31038 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:46.145188 31039 leader_election.cc:336] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.727s (SENT)
I20250902 01:58:46.014986 437 raft_consensus.cc:695] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:46.148871 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.071980 31067 raft_consensus.cc:2391] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:58:46.164238 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 157494us
I20250902 01:58:46.164495 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.169993 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.169903 31038 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:46.171232 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:46.165319 31039 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.171774 31726 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.159289 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.173454 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.174120 30994 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:46.174300 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.171653 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:46.175450 30925 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.072435 372 raft_consensus.cc:3053] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
W20250902 01:58:46.145692 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08003bfc60 after lost signal to thread 30951
I20250902 01:58:46.016659 512 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250902 01:58:46.177831 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 339) took 1116 ms. Trace:
I20250902 01:58:46.187877 30922 rpcz_store.cc:276] 0902 01:58:45.061162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.135007 (+1073845us) service_pool.cc:225] Handling call
0902 01:58:46.177400 (+ 42393us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.177554 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.177581 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.177618 (+ 37us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:46.177658 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.177707 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.177785 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.132352 31728 rpcz_store.cc:276] 0902 01:58:44.992480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.992601 (+ 121us) service_pool.cc:225] Handling call
0902 01:58:45.131614 (+139013us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:45.304751 (+173137us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.304971 (+ 220us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:45.765259 (+460288us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.016741 (+251482us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.016802 (+ 61us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.016865 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.016965 (+ 100us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.017098 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":661,"prepare.queue_time_us":410658,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":76,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":6756,"spinlock_wait_cycles":6768384,"thread_start_us":711132,"threads_started":2,"wal-append.queue_time_us":364092}
I20250902 01:58:46.178773 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 337) took 1328 ms. Trace:
I20250902 01:58:46.180214 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.190359 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.191143 30998 raft_consensus.cc:2391] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:46.128856 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:46.167088 31066 raft_consensus.cc:2887] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:46.177278 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.216629 30994 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:46.133090 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 323) took 1312 ms. Trace:
I20250902 01:58:46.005474 32720 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.133517 368 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:46.227566 368 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.228564 31001 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:46.177083 426 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:46.180735 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1567) took 1395 ms. Trace:
I20250902 01:58:46.148216 32749 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:46.194406 30925 rpcz_store.cc:276] 0902 01:58:44.849950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.131299 (+1281349us) service_pool.cc:225] Handling call
0902 01:58:46.175249 (+ 43950us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.178494 (+ 3245us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.178713 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.166206 389 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
W20250902 01:58:46.180912 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 46471.5us
I20250902 01:58:46.165310 338 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:46.020391 475 consensus_queue.cc:1046] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.157s
I20250902 01:58:46.167853 511 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.205386 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:46.205847 31722 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
W20250902 01:58:46.020275 30992 rpcz_store.cc:269] 0902 01:58:43.812069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:44.912617 (+1100548us) service_pool.cc:225] Handling call
0902 01:58:45.933539 (+1020922us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.207587 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 331) took 1359 ms. Trace:
I20250902 01:58:46.179188 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 770) took 1180 ms. Trace:
I20250902 01:58:46.226191 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1563) took 2432 ms. Trace:
W20250902 01:58:46.284467 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 117556us
I20250902 01:58:46.177789 448 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:46.021328 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1570) took 1065 ms. Trace:
I20250902 01:58:46.160698 314 raft_consensus.cc:695] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.211512 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.226974 30919 rpcz_store.cc:276] 0902 01:58:44.845478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.946009 (+1100531us) service_pool.cc:225] Handling call
0902 01:58:46.005596 (+ 59587us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.026173 (+ 20577us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.046056 (+ 19883us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.056422 (+ 10366us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:46.099714 (+ 43292us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.121541 (+ 21827us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.129310 (+ 7769us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.144791 359 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.045764 32738 consensus_queue.cc:1046] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.158s
I20250902 01:58:46.167106 517 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:46.245958 31069 rpcz_store.cc:276] 0902 01:58:44.785700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.146642 (+1360942us) service_pool.cc:225] Handling call
0902 01:58:46.180344 (+ 33702us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.180485 (+ 141us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.180511 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.180543 (+ 32us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.180570 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.180614 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.180693 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.180716 372 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.175083 30999 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:58:46.263721 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 287367us
I20250902 01:58:46.300531 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.265444 488 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:46.266132 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 127399us
W20250902 01:58:46.301517 30965 leader_election.cc:336] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.682s (SENT)
W20250902 01:58:46.302098 30965 leader_election.cc:336] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.688s (SENT)
W20250902 01:58:46.302871 30965 leader_election.cc:336] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.730s (SENT)
I20250902 01:58:46.303232 30965 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:46.269701 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.270706 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.305097 31001 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:46.272933 508 raft_consensus.cc:3053] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
W20250902 01:58:46.273571 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:46.277343 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.278935 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1516) took 2625 ms. Trace:
W20250902 01:58:46.282012 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.307870 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.308445 30999 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:46.308732 31074 rpcz_store.cc:276] 0902 01:58:43.653397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.794258 (+2140861us) service_pool.cc:225] Handling call
0902 01:58:45.984191 (+189933us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:46.131406 (+147215us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.278688 (+147282us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540036ef50. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.278871 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436360064}
I20250902 01:58:46.056376 344 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.285123 30892 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:46.018023 31724 rpcz_store.cc:276] 0902 01:58:44.819237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.009366 (+190129us) service_pool.cc:225] Handling call
0902 01:58:45.694439 (+685073us) spinlock_profiling.cc:243] Waited 62.7 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:45.694910 (+ 471us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.800453 (+105543us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:45.896794 (+ 96341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144193280}
W20250902 01:58:46.289310 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 140033us
W20250902 01:58:46.289793 31730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.026s sys 0.102s
I20250902 01:58:46.292454 32759 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:46.291988 30998 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 1.
I20250902 01:58:46.322124 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.322719 30998 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.294201 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:46.280037 507 log.cc:927] Time spent T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.341s user 0.000s sys 0.001s
I20250902 01:58:46.163866 386 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.255111 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:46.280808 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08003f2ba0 after lost signal to thread 31025
I20250902 01:58:46.296901 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1564) took 2486 ms. Trace:
I20250902 01:58:46.264504 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1515) took 2850 ms. Trace:
I20250902 01:58:46.341048 31073 rpcz_store.cc:276] 0902 01:58:43.413730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.785900 (+2372170us) service_pool.cc:225] Handling call
0902 01:58:46.005720 (+219820us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.135095 (+129375us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.264292 (+129197us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540003a2d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.264443 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368998144}
W20250902 01:58:46.306226 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.311177 32756 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:46.291440 31071 rpcz_store.cc:276] 0902 01:58:44.952446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.056857 (+104411us) service_pool.cc:225] Handling call
0902 01:58:46.017910 (+961053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.309343 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 325) took 1463 ms. Trace:
I20250902 01:58:46.291962 512 consensus_queue.cc:1046] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250902 01:58:46.313879 30917 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:58:46.309587 31726 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:46.317407 315 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:46.291249 338 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:46.286806 447 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:46.316879 31730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s user 0.026s sys 0.102s
I20250902 01:58:46.307659 32750 consensus_queue.cc:1046] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20250902 01:58:46.281119 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 276714us
I20250902 01:58:46.317782 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 406) took 1358 ms. Trace:
I20250902 01:58:46.358166 31069 rpcz_store.cc:276] 0902 01:58:44.959634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.297229 (+1337595us) service_pool.cc:225] Handling call
0902 01:58:46.317202 (+ 19973us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.317399 (+ 197us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.317437 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.317490 (+ 53us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:46.317538 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.317604 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.317722 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:46.318390 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 173242us
I20250902 01:58:46.286257 31066 rpcz_store.cc:276] 0902 01:58:43.793988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.925050 (+2131062us) service_pool.cc:225] Handling call
0902 01:58:45.925280 (+ 230us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.225870 (+300590us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.226134 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.318805 30995 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250902 01:58:46.366895 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 64051.3us
I20250902 01:58:46.292829 448 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:46.308766 30992 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.325491 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:46.294955 491 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.004s sys 0.008s
I20250902 01:58:46.296345 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:46.269330 32720 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:46.306982 508 raft_consensus.cc:738] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.343009 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:46.259055 32749 raft_consensus.cc:695] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.348394 30996 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:46.415284 30996 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.339380 31070 rpcz_store.cc:276] 0902 01:58:43.810166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.930987 (+2120821us) service_pool.cc:225] Handling call
0902 01:58:45.940301 (+ 9314us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.940496 (+ 195us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:46.147663 (+207167us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.296599 (+148936us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.296649 (+ 50us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.296695 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.296754 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.296852 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":183283,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"thread_start_us":326859,"threads_started":2}
I20250902 01:58:46.349720 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 414) took 1388 ms. Trace:
I20250902 01:58:46.309274 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.416726 31073 rpcz_store.cc:276] 0902 01:58:44.961684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.342020 (+1380336us) service_pool.cc:225] Handling call
0902 01:58:46.349270 (+ 7250us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.349428 (+ 158us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.349453 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.349492 (+ 39us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.349524 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.349573 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.349666 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:46.311885 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 140343us
W20250902 01:58:46.352679 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 67585.4us
I20250902 01:58:46.322753 488 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:46.283725 31723 rpcz_store.cc:276] 0902 01:58:44.992067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.009198 (+ 17131us) service_pool.cc:225] Handling call
0902 01:58:45.790515 (+781317us) spinlock_profiling.cc:243] Waited 62.6 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:46.166582 (+376067us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143965056}
I20250902 01:58:46.356379 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 772) took 1346 ms. Trace:
I20250902 01:58:46.244678 426 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:46.359227 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 79390.1us
I20250902 01:58:46.318569 30994 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:46.405915 32755 raft_consensus.cc:3053] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:46.351590 30920 rpcz_store.cc:276] 0902 01:58:44.846175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.981554 (+1135379us) service_pool.cc:225] Handling call
0902 01:58:45.981777 (+ 223us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.981952 (+ 175us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:46.146864 (+164912us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.309052 (+162188us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.309116 (+ 64us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:46.309155 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.309208 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.309299 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":146289,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":294142,"threads_started":2,"wal-append.queue_time_us":204576}
I20250902 01:58:46.363595 31728 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20250902 01:58:46.359826 31039 leader_election.cc:336] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.747s (SENT)
I20250902 01:58:46.326658 31725 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
W20250902 01:58:46.459168 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 99378.4us
W20250902 01:58:46.367918 30965 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.801s (SENT)
I20250902 01:58:46.368093 31066 raft_consensus.cc:1238] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:46.369542 31001 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.459792 30892 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:46.370771 515 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:46.280664 30921 rpcz_store.cc:276] 0902 01:58:44.847956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.024598 (+1176642us) service_pool.cc:225] Handling call
0902 01:58:46.143264 (+118666us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.172962 (+ 29698us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.176855 (+ 3893us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.183286 (+ 6431us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.187508 (+ 4222us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.207392 (+ 19884us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.207511 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.448274 31730 rpcz_store.cc:276] 0902 01:58:45.009874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.010000 (+ 126us) service_pool.cc:225] Handling call
0902 01:58:45.010314 (+ 314us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.010557 (+ 243us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:45.415882 (+405325us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.356099 (+940217us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.356142 (+ 43us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:46.356183 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.356236 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.356331 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":590,"prepare.queue_time_us":704808,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":44,"raft.queue_time_us":2404,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":3755,"thread_start_us":1324144,"threads_started":2,"wal-append.queue_time_us":925108}
I20250902 01:58:46.372332 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1559) took 3532 ms. Trace:
I20250902 01:58:46.351217 372 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.356067 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:46.368371 448 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.376369 31072 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:46.396104 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.373291 491 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.004s sys 0.009s
I20250902 01:58:46.398140 32756 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:46.374593 514 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.404911 31721 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:58:46.405292 31022 debug-util.cc:398] Leaking SignalData structure 0x7b08003e5840 after lost signal to thread 31099
I20250902 01:58:46.405139 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.463297 523 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:46.364948 32738 consensus_queue.cc:1046] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20250902 01:58:46.405963 31692 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.374178 30993 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 2: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:58:46.430794 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 87878.9us
I20250902 01:58:46.494982 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1566) took 1717 ms. Trace:
I20250902 01:58:46.494867 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:46.495489 31075 rpcz_store.cc:276] 0902 01:58:44.777259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.142368 (+1365109us) service_pool.cc:225] Handling call
0902 01:58:46.175363 (+ 32995us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.494631 (+319268us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.494659 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.494700 (+ 41us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:46.494758 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.494826 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.494923 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":218883,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":1369,"thread_start_us":318852,"threads_started":1}
I20250902 01:58:46.431876 31038 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:46.493996 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 327) took 1647 ms. Trace:
W20250902 01:58:46.498896 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 67117.7us
I20250902 01:58:46.419268 30999 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.436147 31729 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:46.373775 511 raft_consensus.cc:3053] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:46.447827 517 raft_consensus.cc:695] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.450666 32749 consensus_queue.cc:237] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.452140 532 raft_consensus.cc:3053] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:46.501252 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.453039 32759 raft_consensus.cc:3053] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:46.369591 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 340) took 1308 ms. Trace:
I20250902 01:58:46.516023 30917 rpcz_store.cc:276] 0902 01:58:45.061461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.780014 (+718553us) service_pool.cc:225] Handling call
0902 01:58:46.369328 (+589314us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b5400160b54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.369418 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105597824}
W20250902 01:58:46.459795 31039 leader_election.cc:336] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.776s (SENT)
W20250902 01:58:46.517086 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 57329.1us
W20250902 01:58:46.517899 31039 leader_election.cc:336] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.066s (SENT)
W20250902 01:58:46.518600 31039 leader_election.cc:336] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.094s (SENT)
I20250902 01:58:46.518926 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.461809 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1571) took 1167 ms. Trace:
I20250902 01:58:46.519738 31066 rpcz_store.cc:276] 0902 01:58:45.294641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.365944 (+1071303us) service_pool.cc:225] Handling call
0902 01:58:46.367926 (+ 1982us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.461576 (+ 93650us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.461756 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.520630 31075 raft_consensus.cc:1238] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:46.520934 31066 raft_consensus.cc:1238] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:46.460018 30965 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.414980 482 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:46.501420 530 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.463670 31073 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:46.523154 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:46.463994 31068 rpcz_store.cc:276] 0902 01:58:42.840357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.777528 (+2937171us) service_pool.cc:225] Handling call
0902 01:58:45.777765 (+ 237us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:45.777906 (+ 141us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:58:46.097289 (+319383us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.372061 (+274772us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.372104 (+ 43us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:58:46.372143 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.372195 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.372284 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":345659,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":85804,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":535102,"threads_started":3,"wal-append.queue_time_us":300932}
I20250902 01:58:46.351435 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 403) took 1392 ms. Trace:
I20250902 01:58:46.500599 523 raft_consensus.cc:695] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.524542 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:46.524189 31067 rpcz_store.cc:276] 0902 01:58:44.958823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.228773 (+1269950us) service_pool.cc:225] Handling call
0902 01:58:46.287017 (+ 58244us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.308026 (+ 21009us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.310440 (+ 2414us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.311471 (+ 1031us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.320030 (+ 8559us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.330932 (+ 10902us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.349310 (+ 18378us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:46.525022 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 157144us
I20250902 01:58:46.469219 31074 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.525370 535 ts_tablet_manager.cc:927] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:46.447748 31071 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
W20250902 01:58:46.525630 30965 leader_election.cc:336] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.808s (SENT)
I20250902 01:58:46.526021 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.493547 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:46.475365 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.461480 426 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.462133 32755 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:46.368590 314 consensus_queue.cc:237] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.484140 30995 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.524827 523 consensus_queue.cc:237] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.538988 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.539112 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 415) took 1577 ms. Trace:
I20250902 01:58:46.539459 31071 rpcz_store.cc:276] 0902 01:58:44.961875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.359058 (+1397183us) service_pool.cc:225] Handling call
0902 01:58:46.402519 (+ 43461us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.533883 (+131364us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.539057 (+ 5174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:46.541869 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.307523 437 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.477871 515 raft_consensus.cc:695] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:58:46.489070 519 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250902 01:58:46.480109 32756 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:46.490386 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.500298 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:46.563244 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 63015.3us
I20250902 01:58:46.503052 31726 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:46.504114 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 98225.9us
I20250902 01:58:46.484167 372 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.512621 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:46.482887 508 consensus_queue.cc:260] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.522511 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:46.503597 531 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:46.515813 32759 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.527256 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:46.441888 30992 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:46.636308 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.499423 30918 rpcz_store.cc:276] 0902 01:58:44.846750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.986643 (+1139893us) service_pool.cc:225] Handling call
0902 01:58:46.044384 (+ 57741us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:46.044542 (+ 158us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:46.122103 (+ 77561us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:46.122156 (+ 53us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:46.132267 (+ 10111us) log.cc:844] Serialized 26 byte log entry
0902 01:58:46.317979 (+185712us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.349255 (+ 31276us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.356425 (+ 7170us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.356489 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:46.483953 (+127464us) raft_consensus.cc:1683] finished
0902 01:58:46.493820 (+ 9867us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.493935 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83395,"prepare.queue_time_us":32877,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":113176,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":22295,"spinlock_wait_cycles":2780928,"thread_start_us":174729,"threads_started":1,"wal-append.queue_time_us":307467}
I20250902 01:58:46.481387 30919 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
W20250902 01:58:46.637318 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 172097us
I20250902 01:58:46.557264 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.646551 30994 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:46.648304 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.572945 30998 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
W20250902 01:58:46.650257 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.650942 30998 raft_consensus.cc:1238] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:46.503582 31022 debug-util.cc:398] Leaking SignalData structure 0x7b080034ac00 after lost signal to thread 31754
W20250902 01:58:46.532696 31039 leader_election.cc:336] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.189s (SENT)
I20250902 01:58:46.656476 31072 raft_consensus.cc:1238] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:46.657281 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 124546us
I20250902 01:58:46.578608 31722 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:46.586376 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:46.526862 30965 leader_election.cc:336] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.858s (SENT)
I20250902 01:58:46.662369 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.665199 31724 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:46.544700 31075 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:46.572839 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.565240 31692 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.668627 31001 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250902 01:58:46.669682 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 104345us
I20250902 01:58:46.670447 31001 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.641803 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.644271 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.534838 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:46.704501 30999 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:46.717816 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.721752 475 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:46.722903 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.555915 30995 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:46.639240 30924 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:46.537101 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:46.719230 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.725735 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 188613us
I20250902 01:58:46.573881 32720 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.580752 32755 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.533939 315 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:58:46.501513 338 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:46.503897 359 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:46.533941 368 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.743989 368 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:46.558552 454 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.543s user 0.006s sys 0.010s
I20250902 01:58:46.746978 30993 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:58:46.533564 488 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.750996 530 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.751952 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 413) took 1790 ms. Trace:
I20250902 01:58:46.752449 31074 rpcz_store.cc:276] 0902 01:58:44.961456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.309895 (+1348439us) service_pool.cc:225] Handling call
0902 01:58:46.469062 (+159167us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.525616 (+ 56554us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.751560 (+225944us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54000406d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.751764 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365328640}
W20250902 01:58:46.452366 489 consensus_meta.cc:287] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.526s user 0.009s sys 0.001s
I20250902 01:58:46.753784 30918 raft_consensus.cc:1238] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:46.755216 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.755810 447 raft_consensus.cc:3053] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:46.756248 447 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:58:46.757411 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.757637 447 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.759234 447 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.575778 508 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:46.504359 511 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:58:46.768819 31727 consensus_meta.cc:287] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.724s user 0.010s sys 0.006s
I20250902 01:58:46.603163 515 consensus_queue.cc:237] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.521669 517 consensus_queue.cc:237] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.769637 338 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:46.770238 338 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.571791 522 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:46.772329 522 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:46.515321 525 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.773981 30997 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:46.523324 526 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:46.775517 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 281) took 1008 ms. Trace:
I20250902 01:58:46.775974 30997 rpcz_store.cc:276] 0902 01:58:45.766988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.933510 (+166522us) service_pool.cc:225] Handling call
0902 01:58:46.775275 (+841765us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54002ad0d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.775444 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1707512704}
W20250902 01:58:46.612742 527 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.208s user 0.003s sys 0.000s
I20250902 01:58:46.769573 511 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.480461 529 consensus_queue.cc:1046] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20250902 01:58:46.778952 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 330) took 1931 ms. Trace:
I20250902 01:58:46.779738 511 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.780190 529 raft_consensus.cc:3053] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:46.780563 529 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.779446 30924 rpcz_store.cc:276] 0902 01:58:44.847680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.015702 (+1168022us) service_pool.cc:225] Handling call
0902 01:58:46.016010 (+ 308us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.724497 (+708487us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.778670 (+ 54173us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b540026fcd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:46.778878 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1432032640}
W20250902 01:58:46.781383 31039 leader_election.cc:336] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.828s (SENT)
I20250902 01:58:46.781761 529 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.597298 531 raft_consensus.cc:695] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:46.514691 532 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.784994 532 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.786303 532 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:46.544237 533 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20250902 01:58:46.612262 535 tablet_copy_client.cc:323] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:46439
I20250902 01:58:46.681289 512 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:46.789536 512 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:46.683168 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 156331us
W20250902 01:58:46.792137 30965 leader_election.cc:336] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.901s (SENT)
I20250902 01:58:46.687707 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.792254 544 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20250902 01:58:46.793195 30919 raft_consensus.cc:2391] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:46.688141 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.688539 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:46.691133 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:46.694783 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.698518 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.692793 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.723120 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:46.724455 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.795324 30923 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:46.648383 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:46.798334 512 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:46.799818 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.773976 522 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.801473 482 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:46.802491 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:46.801489 30965 leader_election.cc:336] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.910s (SENT)
I20250902 01:58:46.803090 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:46.804754 30965 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.910s (SENT)
I20250902 01:58:46.805189 30965 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.805322 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.733806 31692 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.806071 32720 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:58:46.806695 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 133331us
I20250902 01:58:46.806941 32720 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.726581 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.769793 31727 raft_consensus.cc:2408] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:46.809123 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 361) took 1018 ms. Trace:
I20250902 01:58:46.792310 531 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.727226 475 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:58:46.805418 32755 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.808172 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.739193 31693 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:46.748337 521 log.cc:927] Time spent T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.323s user 0.010s sys 0.000s
W20250902 01:58:46.841780 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.807452 32738 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:58:46.842805 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 277151us
W20250902 01:58:46.844727 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 105753us
I20250902 01:58:46.809505 31727 rpcz_store.cc:276] 0902 01:58:45.790318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.925057 (+134739us) service_pool.cc:225] Handling call
0902 01:58:46.809038 (+883981us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:46.852195 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.841948 475 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:46.735685 32759 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.856113 31725 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:46.857779 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.859388 344 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:46.859795 344 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:46.860127 344 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:46.860939 525 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.861941 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.863982 475 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.865216 529 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:46.877110 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.877625 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.877820 30923 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:46.878762 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.879884 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.881014 30965 leader_election.cc:336] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.127s (SENT)
I20250902 01:58:46.884460 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:46.886078 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.886868 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.887833 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.888146 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.889353 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.890045 30921 raft_consensus.cc:2391] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:46.890337 347 consensus_meta.cc:287] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.710s user 0.009s sys 0.012s
I20250902 01:58:46.890781 347 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:46.892207 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.894438 426 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.896041 30925 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:46.896932 426 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:46.899629 31074 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.904168 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.905673 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:46.907716 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.909224 31000 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:46.910442 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 285) took 1142 ms. Trace:
I20250902 01:58:46.910760 31000 rpcz_store.cc:276] 0902 01:58:45.767732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.818591 (+ 50859us) service_pool.cc:225] Handling call
0902 01:58:46.910235 (+1091644us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54001af9d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:46.910369 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1684835584}
I20250902 01:58:46.875391 32756 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.914638 552 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:46.916867 32756 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:46.916575 372 consensus_queue.cc:1046] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:46.917829 372 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:46.918166 372 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.918422 338 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.921710 31075 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:46.921957 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:46.923436 31038 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:46.924692 556 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:46.925477 556 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:46.924692 30923 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:46.925640 31038 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:46.926383 551 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250902 01:58:46.926637 556 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:46.929175 508 consensus_queue.cc:1046] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:46.930576 550 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:46.930814 508 raft_consensus.cc:3053] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:46.931113 550 raft_consensus.cc:695] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:46.931187 508 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:46.932116 550 consensus_queue.cc:237] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.932463 508 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.933781 508 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:46.931457 338 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:46.931273 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 123312us
I20250902 01:58:46.935360 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:46.938609 347 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:46.938807 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 157442us
W20250902 01:58:46.939733 31039 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.831s (SENT)
W20250902 01:58:46.941002 31039 leader_election.cc:336] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.036s (SENT)
I20250902 01:58:46.941463 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.943862 31692 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:46.943213 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 86872us
I20250902 01:58:46.949280 30923 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:46.938722 32759 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:46.953687 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:46.956048 372 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:46.957381 372 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:46.957487 547 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:46.959251 30922 raft_consensus.cc:2408] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:46.959995 31692 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.961057 447 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:46.964223 31039 leader_election.cc:336] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.063s (SENT)
I20250902 01:58:46.964732 31039 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.965564 556 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:46.966089 31039 leader_election.cc:336] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.075s (SENT)
I20250902 01:58:46.972527 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:46.979403 554 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20250902 01:58:46.973235 31692 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:46.981768 30918 raft_consensus.cc:2408] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:46.986568 534 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:46.987414 534 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:46.987843 534 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:46.971905 511 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:46.990833 525 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:46.995957 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:46.999213 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:46.999878 428 consensus_queue.cc:1046] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:47.000589 522 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.001096 437 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:47.002101 437 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:47.002417 437 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.005277 30923 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:47.007709 30919 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
W20250902 01:58:47.010793 448 consensus_meta.cc:287] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.005s
I20250902 01:58:47.010548 344 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.011277 448 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.013950 438 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20250902 01:58:46.989431 30965 leader_election.cc:336] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.133s (SENT)
I20250902 01:58:47.021446 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:46.989316 31038 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.023097 531 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:47.023455 522 consensus_queue.cc:1046] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:47.023589 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.024287 347 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:47.024839 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 416) took 2028 ms. Trace:
I20250902 01:58:47.024941 347 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.025375 31038 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.025246 31069 rpcz_store.cc:276] 0902 01:58:44.962093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.359126 (+1397033us) service_pool.cc:225] Handling call
0902 01:58:46.415449 (+ 56323us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:46.415894 (+ 445us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:58:46.437905 (+ 22011us) log.cc:844] Serialized 14 byte log entry
0902 01:58:46.936215 (+498310us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:46.950986 (+ 14771us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:46.968585 (+ 17599us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:46.979992 (+ 11407us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:46.990009 (+ 10017us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":20980,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":33674,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":4657,"thread_start_us":442780,"threads_started":2,"wal-append.queue_time_us":165071}
I20250902 01:58:47.026571 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.000602 428 consensus_queue.cc:1230] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250902 01:58:47.024335 522 raft_consensus.cc:3053] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:47.027920 522 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.027984 347 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:47.028430 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 62392us
I20250902 01:58:47.023898 531 raft_consensus.cc:695] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.029747 531 consensus_queue.cc:237] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.029166 522 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.030843 338 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:47.030946 556 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:47.030822 547 consensus_queue.cc:1046] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:58:47.031750 426 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
W20250902 01:58:47.024478 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.036165 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.042897 31070 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.043573 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.044246 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1572) took 1292 ms. Trace:
I20250902 01:58:47.044181 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.003244 437 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.044713 31070 rpcz_store.cc:276] 0902 01:58:45.752206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.416574 (+664368us) service_pool.cc:225] Handling call
0902 01:58:46.458620 (+ 42046us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:47.043343 (+584723us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:47.044011 (+ 668us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54001e82d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:47.044201 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1334083328}
I20250902 01:58:47.041863 347 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.053485 556 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.054454 426 raft_consensus.cc:695] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.051324 30922 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
W20250902 01:58:47.057842 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 94320.3us
I20250902 01:58:47.049566 525 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250902 01:58:47.059927 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 87420us
W20250902 01:58:47.060796 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.061771 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:47.064179 389 consensus_meta.cc:287] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.795s user 0.003s sys 0.011s
I20250902 01:58:47.064332 522 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.064414 556 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.065454 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.064929 30926 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:47.101691 344 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:47.111835 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:47.111788 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 98482.1us
I20250902 01:58:47.117977 31692 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:47.112200 338 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.113780 389 consensus_queue.cc:1046] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250902 01:58:47.125466 489 consensus_queue.cc:1046] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250902 01:58:47.125885 561 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20250902 01:58:47.115057 437 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.131011 347 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:47.131600 347 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.132454 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.132817 448 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:47.133842 347 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.133486 426 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.135063 344 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:47.135424 344 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:47.135823 344 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.123211 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.131807 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.136202 338 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.148324 30917 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.152639 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:47.153165 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.153838 482 raft_consensus.cc:3053] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:47.154072 31071 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20250902 01:58:47.154217 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 110786us
I20250902 01:58:47.154270 482 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.145254 489 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250902 01:58:47.155478 482 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.156445 534 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.156960 454 raft_consensus.cc:991] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: : Instructing follower 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:47.157080 482 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.157559 454 raft_consensus.cc:1079] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Signalling peer 4372b4c4c866431faa1bf3e91fd47a88 to start an election
I20250902 01:58:47.157768 517 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250902 01:58:47.160440 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 48786.1us
W20250902 01:58:47.183178 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.158974 32720 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.009s
I20250902 01:58:47.148331 488 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:47.195176 30917 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
W20250902 01:58:47.196177 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.194541 570 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:47.197539 514 consensus_meta.cc:287] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.658s user 0.010s sys 0.010s
I20250902 01:58:47.203893 31070 raft_consensus.cc:1273] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250902 01:58:47.209779 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:47.207938 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 55534.4us
W20250902 01:58:47.220644 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.232019 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:47.301144 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 179092us
I20250902 01:58:47.208326 31070 pending_rounds.cc:85] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 1
I20250902 01:58:47.236877 514 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:47.274457 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 114590us
W20250902 01:58:47.289491 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 78732.6us
I20250902 01:58:47.208614 31693 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:47.204201 529 raft_consensus.cc:3053] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:47.291687 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:47.299969 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:47.305603 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.303349 31070 pending_rounds.cc:107] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250902 01:58:47.308776 31070 raft_consensus.cc:2887] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:47.314214 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:47.315574 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 107048us
I20250902 01:58:47.303522 31071 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
W20250902 01:58:47.317389 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.308329 529 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.318578 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.315758 31692 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.318773 30917 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.335088 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.330437 573 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:47.343477 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:47.343845 30965 leader_election.cc:336] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.599s (SENT)
I20250902 01:58:47.351734 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.352862 30924 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 2: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:47.353853 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.361156 30920 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:47.363945 30965 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.362952 30923 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:47.365913 31075 raft_consensus.cc:1273] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250902 01:58:47.347779 573 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.365693 526 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.368441 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.336010 529 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.369077 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.368937 573 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.348119 534 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:47.369823 529 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:47.370682 577 log.cc:927] Time spent T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250902 01:58:47.375150 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.369035 30926 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
W20250902 01:58:47.383639 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.383989 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.384790 517 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:47.385811 30918 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:47.386791 488 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:47.386703 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:58:47.393721 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 101821us
W20250902 01:58:47.394811 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 53284.2us
I20250902 01:58:47.395277 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.399791 528 raft_consensus.cc:3053] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.400156 528 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.400329 31001 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:47.400456 31000 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250902 01:58:47.396260 570 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:47.401010 528 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.401594 570 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.401995 528 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.397600 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:47.402631 570 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.404590 30889 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:47.387558 525 raft_consensus.cc:2802] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250902 01:58:47.409210 31695 leader_election.cc:336] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.639s (SENT)
I20250902 01:58:47.388458 508 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250902 01:58:47.412739 31692 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.396123 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.413163 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.426796 32759 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:47.428650 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.433934 30921 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.433782 344 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.435259 31070 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.407677 517 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:47.436160 517 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:47.436367 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.436502 517 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:47.437676 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 51148.6us
I20250902 01:58:47.407997 525 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:47.446281 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 107283us
I20250902 01:58:47.449193 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:47.453994 30994 raft_consensus.cc:1238] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
W20250902 01:58:47.449968 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45570us
I20250902 01:58:47.454809 31074 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250902 01:58:47.460376 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 53217.4us
I20250902 01:58:47.461845 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.464740 31001 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
W20250902 01:58:47.466993 30965 leader_election.cc:336] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.638s (SENT)
I20250902 01:58:47.449717 32738 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.447645 525 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.454509 32755 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:47.471712 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.455004 534 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:47.472573 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:47.473464 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.461746 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.475679 31039 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.455005 515 consensus_queue.cc:1046] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250902 01:58:47.477198 515 raft_consensus.cc:3053] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.477514 515 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.478304 515 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.453773 389 consensus_queue.cc:1046] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20250902 01:58:47.455042 32759 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.454782 556 consensus_queue.cc:1046] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
W20250902 01:58:47.486443 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 77385us
W20250902 01:58:47.492626 30965 leader_election.cc:336] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.197s (SENT)
I20250902 01:58:47.493132 30965 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.479365 30919 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:47.479463 515 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.515970 30997 raft_consensus.cc:1273] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:47.484030 344 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:47.491853 550 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.509235 32759 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.509591 31039 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.494483 426 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:47.509819 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:47.510212 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 48590.7us
W20250902 01:58:47.514618 30967 leader_election.cc:336] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.619s (SENT)
I20250902 01:58:47.492213 30992 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.475409 508 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
W20250902 01:58:47.527065 30965 leader_election.cc:336] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.236s (SENT)
I20250902 01:58:47.524155 30921 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:58:47.507006 488 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.532032 31692 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.537915 31725 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:47.539889 437 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:58:47.532531 570 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.532032 426 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.527778 32755 consensus_queue.cc:1046] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250902 01:58:47.550119 32755 raft_consensus.cc:3053] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:47.550545 32755 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.552012 571 raft_consensus.cc:3053] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:47.552402 571 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.556579 532 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:47.556977 532 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.557740 532 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.553375 571 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.561978 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.562613 571 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.562732 32759 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:47.563169 32759 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:47.558841 532 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.563577 32759 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:47.569056 30965 leader_election.cc:336] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.244s (SENT)
W20250902 01:58:47.569245 582 log.cc:927] Time spent T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.120s user 0.000s sys 0.002s
W20250902 01:58:47.570005 30965 leader_election.cc:336] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.858s (SENT)
I20250902 01:58:47.566329 526 raft_consensus.cc:3053] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.571722 526 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:47.572774 31038 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.572618 526 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.573850 30823 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44296 (request call id 351) took 8726 ms. Trace:
I20250902 01:58:47.573917 526 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.574663 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.575762 426 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.563305 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.580623 508 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:58:47.580993 508 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:47.581347 508 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:58:47.587890 30965 leader_election.cc:336] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.619s (SENT)
I20250902 01:58:47.588232 30965 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:47.589066 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.584960 514 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:47.591305 32738 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:47.574312 30823 rpcz_store.cc:276] 0902 01:58:38.847091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:38.847252 (+ 161us) service_pool.cc:225] Handling call
0902 01:58:47.573560 (+8726308us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:45.542232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:45.554373 (+ 12141us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:45.554416 (+ 43us) write_op.cc:435] Acquired schema lock
0902 01:58:45.554440 (+ 24us) tablet.cc:660] Decoding operations
0902 01:58:45.555109 (+ 669us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:45.555197 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:45.555233 (+ 36us) tablet.cc:683] Acquiring locks for 3 operations
0902 01:58:45.555633 (+ 400us) tablet.cc:699] Row locks acquired
0902 01:58:45.555656 (+ 23us) write_op.cc:260] PREPARE: finished
0902 01:58:45.555817 (+ 161us) write_op.cc:270] Start()
0902 01:58:45.555957 (+ 140us) write_op.cc:276] Timestamp: P: 1756778325555762 usec, L: 0
0902 01:58:45.555978 (+ 21us) op_driver.cc:348] REPLICATION: starting
0902 01:58:45.661534 (+105556us) log.cc:844] Serialized 1307 byte log entry
0902 01:58:46.607490 (+945956us) spinlock_profiling.cc:243] Waited 8.08 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:46.908484 (+300994us) op_driver.cc:464] REPLICATION: finished
0902 01:58:47.049434 (+140950us) write_op.cc:301] APPLY: starting
0902 01:58:47.058873 (+ 9439us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:47.070148 (+ 11275us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:47.070195 (+ 47us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:47.161989 (+ 91794us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:47.251176 (+ 89187us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0902 01:58:47.251282 (+ 106us) write_op.cc:312] APPLY: finished
0902 01:58:47.251718 (+ 436us) log.cc:844] Serialized 35 byte log entry
0902 01:58:47.251963 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:47.301007 (+ 49044us) write_op.cc:454] Released schema lock
0902 01:58:47.301072 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:47.301491 (+ 419us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:47.301518 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":61834,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":110692,"apply.run_cpu_time_us":13756,"apply.run_wall_time_us":264740,"num_ops":3,"prepare.queue_time_us":652362,"prepare.run_cpu_time_us":16758,"prepare.run_wall_time_us":939136,"raft.queue_time_us":129362,"raft.run_cpu_time_us":9448,"raft.run_wall_time_us":40193,"replication_time_us":1352263,"spinlock_wait_cycles":18587904,"thread_start_us":1153489,"threads_started":4,"wal-append.queue_time_us":698385}]]}
I20250902 01:58:47.593199 488 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:47.605347 30965 leader_election.cc:336] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.075s (SENT)
I20250902 01:58:47.608676 31038 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:47.609445 534 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:47.609771 534 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:47.610103 31038 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:47.610131 534 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.611392 338 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.601666 522 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:58:47.618817 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 87336.1us
I20250902 01:58:47.601439 514 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:47.622088 570 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:47.623418 389 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:47.585310 32755 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.626487 514 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.624058 389 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.635929 32755 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.630098 522 raft_consensus.cc:695] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:47.644170 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.648912 488 consensus_queue.cc:1046] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:47.649827 488 consensus_queue.cc:1230] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250902 01:58:47.650259 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:47.650213 488 raft_consensus.cc:3053] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:47.650907 30997 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:47.651144 488 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.648905 30819 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 2 to 4, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 4 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:47.664036 30819 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: term changed from 1 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:58:47.665702 30889 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:47.666749 30965 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.670526 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:47.671785 585 log.cc:927] Time spent T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20250902 01:58:47.671743 372 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:47.672119 389 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.668533 532 consensus_queue.cc:1046] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250902 01:58:47.674443 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.674898 534 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:47.678211 31695 leader_election.cc:336] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.903s (SENT)
I20250902 01:58:47.670440 522 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.680466 534 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.676290 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.668020 515 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:47.654991 488 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.675729 347 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:47.696764 515 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:47.696991 347 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:47.697233 488 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.697372 347 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.698347 31727 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:47.700595 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.701368 31074 raft_consensus.cc:1238] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.701498 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.702003 31069 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.702347 30992 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:47.674849 532 raft_consensus.cc:3053] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:47.703013 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.703231 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.670647 32759 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.705475 31071 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.705600 32759 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.687793 30964 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:47.707371 482 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.707840 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.676700 525 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.708061 482 consensus_queue.cc:1046] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:58:47.708808 489 raft_consensus.cc:3053] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.709242 489 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.710883 31011 tablet_copy_service.cc:140] P 975ecd397f5146409baac87e56a302e9: Received BeginTabletCopySession request for tablet 92f16b8a15144fcf8807608bdee71123 from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:54174)
I20250902 01:58:47.711529 31011 tablet_copy_service.cc:161] P 975ecd397f5146409baac87e56a302e9: Beginning new tablet copy session on tablet 92f16b8a15144fcf8807608bdee71123 from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:54174: session id = 01f6731ce5044a6a948a5ae04da80e6f-92f16b8a15144fcf8807608bdee71123
I20250902 01:58:47.712251 30994 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.713981 31000 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.714924 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.715426 31001 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.683657 588 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.695688 517 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.725346 517 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:47.726068 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:47.726915 31069 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.727782 30997 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:47.730225 31001 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:47.730268 525 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:47.710331 489 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.731878 489 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.697229 515 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.725114 347 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.737478 347 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.703351 532 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.738400 31070 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:47.739487 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:47.740571 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.740888 588 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:47.741813 588 raft_consensus.cc:3053] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.741894 31069 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:47.742137 588 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.740058 30819 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f reported cstate change: config changed from index 4 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:58:47.738830 532 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.743188 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:47.743700 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 50336.3us
W20250902 01:58:47.744509 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.744036 588 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.745231 588 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.745182 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.691356 514 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.748731 508 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.749433 514 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.750897 508 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:47.751374 532 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:47.751986 30992 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.752569 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.745404 32759 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:47.755664 32759 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:58:47.759567 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.760623 30918 raft_consensus.cc:1238] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.762274 31069 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:47.762470 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 86249.5us
W20250902 01:58:47.759730 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.766675 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:47.776607 30889 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:47.777979 32755 consensus_queue.cc:1046] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:47.778322 508 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.778573 30889 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.778939 32755 raft_consensus.cc:3053] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.779454 32755 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.780471 32755 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.782147 32755 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.759043 32759 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:47.771932 31695 leader_election.cc:336] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.007s (SENT)
I20250902 01:58:47.789541 31695 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.790460 475 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.793591 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:47.775975 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.769083 31011 tablet_copy_source_session.cc:215] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:58:47.799105 517 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:47.804109 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:47.805112 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:47.806824 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 60104.2us
I20250902 01:58:47.807960 31074 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:47.808442 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.809031 31725 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
W20250902 01:58:47.809144 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.810822 31725 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.812350 30918 raft_consensus.cc:1238] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
I20250902 01:58:47.813786 475 consensus_queue.cc:1046] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:58:47.814770 475 raft_consensus.cc:3053] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:58:47.815114 475 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.815827 475 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.816928 475 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:47.798849 515 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.823248 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.824782 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.825136 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.825469 31721 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:47.825766 31725 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:47.826541 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.827311 31723 raft_consensus.cc:2391] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:47.830324 31727 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:47.820699 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:47.835583 489 consensus_queue.cc:1046] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:47.836426 489 raft_consensus.cc:3053] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.836696 489 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.837378 489 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.838510 489 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.844517 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.845304 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.845518 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.845891 588 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.847606 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:47.850571 588 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:47.851001 588 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.851724 532 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:47.851945 515 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:47.851421 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.853852 588 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.855901 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:47.858601 31727 raft_consensus.cc:2374] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2. Re-sending same reply.
I20250902 01:58:47.859001 30918 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:47.861730 489 consensus_queue.cc:1046] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:58:47.862141 347 consensus_queue.cc:1046] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250902 01:58:47.862758 488 raft_consensus.cc:3053] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.862963 30918 raft_consensus.cc:1215] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20250902 01:58:47.863108 488 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.863369 347 raft_consensus.cc:3053] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.864040 30921 raft_consensus.cc:1238] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:47.863868 488 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.864769 488 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.863373 30918 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:47.868489 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.863883 347 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:47.869385 30921 raft_consensus.cc:2408] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:47.869853 347 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.870488 30921 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:38796
I20250902 01:58:47.871235 30921 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250902 01:58:47.871649 30921 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:47.871991 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.872570 31727 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.877072 347 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.877323 489 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:47.880168 30917 raft_consensus.cc:1215] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20250902 01:58:47.880347 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.880648 30917 raft_consensus.cc:1238] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:47.883028 31727 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:47.886505 30920 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:47.886612 522 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:47.887338 522 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:47.888010 31039 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.889199 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:47.889664 30992 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:47.889669 522 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:47.890992 522 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:47.891664 522 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:47.892035 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.893952 30992 raft_consensus.cc:1273] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250902 01:58:47.894312 30992 pending_rounds.cc:85] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 1
I20250902 01:58:47.894552 30992 pending_rounds.cc:107] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250902 01:58:47.894807 30992 raft_consensus.cc:2887] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:47.889230 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:47.896045 30920 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.896675 30917 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:47.898100 32755 consensus_queue.cc:1046] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:47.898953 573 raft_consensus.cc:3053] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:47.899255 573 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:47.899797 31071 raft_consensus.cc:1238] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:47.899977 573 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.901196 573 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.903051 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.903231 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.903992 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.905186 517 consensus_queue.cc:1046] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:47.906240 532 raft_consensus.cc:3053] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:47.906594 532 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:47.907375 532 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:47.908936 532 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:47.914292 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.914942 31726 raft_consensus.cc:2374] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2. Re-sending same reply.
I20250902 01:58:47.915478 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.918738 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.919337 30997 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:47.921500 30965 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.923339 30992 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250902 01:58:47.923740 30992 pending_rounds.cc:85] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 1
I20250902 01:58:47.923941 30992 pending_rounds.cc:107] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250902 01:58:47.924171 30992 raft_consensus.cc:2887] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:47.924654 31725 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:47.928470 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.928536 31721 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:47.928830 31071 raft_consensus.cc:1273] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:47.929083 31000 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:47.930927 30965 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:47.915612 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.932044 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.934454 31000 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:47.920572 30920 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:47.942008 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.891036 30921 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:47.952927 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:47.956722 31723 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:47.962383 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.942844 488 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:47.960861 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.963011 30919 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 7. Ops: []
I20250902 01:58:47.962841 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:47.956027 517 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20250902 01:58:47.993983 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:47.971246 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:47.994727 31726 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:47.999653 30819 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40172 (request call id 251) took 2941 ms. Trace:
W20250902 01:58:48.010596 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 79754.4us
I20250902 01:58:47.996764 30921 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:48.012202 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.000384 30992 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:48.012832 31726 raft_consensus.cc:2408] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:47.999996 30819 rpcz_store.cc:276] 0902 01:58:45.054520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:45.063141 (+ 8621us) service_pool.cc:225] Handling call
0902 01:58:47.996191 (+2933050us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:47.812277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:47.812912 (+ 635us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:47.812963 (+ 51us) write_op.cc:435] Acquired schema lock
0902 01:58:47.812980 (+ 17us) tablet.cc:660] Decoding operations
0902 01:58:47.813560 (+ 580us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:47.813653 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:47.813857 (+ 204us) tablet.cc:683] Acquiring locks for 3 operations
0902 01:58:47.814265 (+ 408us) tablet.cc:699] Row locks acquired
0902 01:58:47.814281 (+ 16us) write_op.cc:260] PREPARE: finished
0902 01:58:47.814569 (+ 288us) write_op.cc:270] Start()
0902 01:58:47.814714 (+ 145us) write_op.cc:276] Timestamp: P: 1756778327814496 usec, L: 0
0902 01:58:47.814745 (+ 31us) op_driver.cc:348] REPLICATION: starting
0902 01:58:47.815329 (+ 584us) log.cc:844] Serialized 1365 byte log entry
0902 01:58:47.925614 (+110285us) op_driver.cc:464] REPLICATION: finished
0902 01:58:47.927113 (+ 1499us) write_op.cc:301] APPLY: starting
0902 01:58:47.927186 (+ 73us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:47.927303 (+ 117us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:47.927405 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:47.927900 (+ 495us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:47.928075 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0902 01:58:47.928124 (+ 49us) write_op.cc:312] APPLY: finished
0902 01:58:47.929324 (+ 1200us) log.cc:844] Serialized 35 byte log entry
0902 01:58:47.929469 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:47.929547 (+ 78us) write_op.cc:454] Released schema lock
0902 01:58:47.929581 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":1280,"apply.run_cpu_time_us":1761,"apply.run_wall_time_us":2768,"num_ops":3,"prepare.queue_time_us":54425,"prepare.run_cpu_time_us":2939,"prepare.run_wall_time_us":9441,"replication_time_us":110711,"thread_start_us":47688,"threads_started":2}]]}
I20250902 01:58:48.023880 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.024365 31730 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:48.025116 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.027932 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.028267 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.028844 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.031879 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.031983 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:48.032639 30922 raft_consensus.cc:2408] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250902 01:58:48.033334 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.033725 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.034361 30925 raft_consensus.cc:2374] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2. Re-sending same reply.
I20250902 01:58:48.035297 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.036502 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:48.037074 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.032500 30964 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:48.033350 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:48.040021 30997 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:48.041806 31730 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:48.042480 30917 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:48.043180 30920 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:48.045869 31724 raft_consensus.cc:3053] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:48.046229 31724 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:48.046913 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.053313 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:48.044781 30920 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:48.055899 30923 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:48.056679 31074 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250902 01:58:48.035892 30925 raft_consensus.cc:2374] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3. Re-sending same reply.
I20250902 01:58:48.017685 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:48.066299 30923 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:48.065001 31692 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.067989 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:48.070094 30920 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:48.071120 30917 raft_consensus.cc:1273] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:48.070380 30923 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:48.069343 508 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20250902 01:58:48.046949 31724 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:48.134220 606 log.cc:927] Time spent T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250902 01:58:48.146344 30922 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:48.134353 30920 raft_consensus.cc:1273] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:48.130373 32738 consensus_queue.cc:1046] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250902 01:58:48.128443 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:48.130699 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 65805.4us
I20250902 01:58:48.139462 30965 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:48.144374 604 consensus_queue.cc:1046] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 8, Time since last communication: 0.011s
I20250902 01:58:48.140202 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:48.141094 31039 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.143514 30824 catalog_manager.cc:5582] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:48.142282 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.057564 607 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:48.137219 31722 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:48.071427 588 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:48.130762 30924 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:48.152701 30824 catalog_manager.cc:5582] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:48.153370 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.129336 609 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:48.157744 30824 catalog_manager.cc:5582] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:58:48.071660 389 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250902 01:58:48.071261 372 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20250902 01:58:48.159353 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.159106 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.163209 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:48.145455 475 raft_consensus.cc:2762] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250902 01:58:48.144444 482 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:48.163962 31070 raft_consensus.cc:2374] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3. Re-sending same reply.
I20250902 01:58:48.172626 32738 consensus_queue.cc:1230] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250902 01:58:48.166817 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.173455 31729 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:58:48.173871 31729 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:48.174151 372 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:48.168061 30824 catalog_manager.cc:5582] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:58:48.177685 30824 catalog_manager.cc:5582] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 2, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194), now has a pending config: VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193), VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194), VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195), NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:48.180428 31067 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:48.182071 31725 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:58:48.185688 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.171418 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:48.174765 31729 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.188544 31729 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:48.156414 514 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:48.201061 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.207017 31001 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:48.170665 482 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:48.208667 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:48.156615 573 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20250902 01:58:48.190330 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:48.213958 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 63035.1us
I20250902 01:58:48.215236 30917 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:48.218413 30994 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250902 01:58:48.205260 525 raft_consensus.cc:3053] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:48.201658 32759 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:48.196228 488 raft_consensus.cc:695] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.218840 30994 pending_rounds.cc:85] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 1
I20250902 01:58:48.268867 30994 pending_rounds.cc:107] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250902 01:58:48.269171 30994 raft_consensus.cc:2887] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:48.206894 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.270766 31730 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:48.220556 426 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:48.263007 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:48.263911 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 47489.2us
I20250902 01:58:48.212046 482 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:48.262389 604 consensus_queue.cc:1230] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer 01f6731ce5044a6a948a5ae04da80e6f log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20250902 01:58:48.266345 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.267222 31692 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:48.213295 347 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:48.271554 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:48.271734 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 113163us
I20250902 01:58:48.280898 31000 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:48.210391 508 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.210305 344 consensus_queue.cc:1046] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20250902 01:58:48.206655 550 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:48.264781 573 consensus_queue.cc:1230] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250902 01:58:48.213850 571 consensus_queue.cc:1046] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20250902 01:58:48.195574 609 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.265712 525 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:48.297151 31000 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:48.302647 31722 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:48.304261 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.282413 426 raft_consensus.cc:695] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.312521 30926 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:48.314499 31723 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:48.287952 347 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.282907 570 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250902 01:58:48.287153 508 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.316274 550 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:48.391861 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 124717us
I20250902 01:58:48.394459 607 raft_consensus.cc:695] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.398810 607 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:48.395792 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 178121us
I20250902 01:58:48.408563 31001 raft_consensus.cc:1273] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:48.324519 32738 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
W20250902 01:58:48.408423 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 255217us
I20250902 01:58:48.416358 31039 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:48.429481 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:48.412325 30993 raft_consensus.cc:1273] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:48.433930 31726 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:48.434154 30995 raft_consensus.cc:1273] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250902 01:58:48.434301 31726 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:48.434687 31722 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:48.434940 629 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:48.435134 620 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:48.435052 31726 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.425015 532 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.436589 629 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:48.436956 532 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:48.380389 618 consensus_queue.cc:1046] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20250902 01:58:48.439982 31730 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:48.416603 588 raft_consensus.cc:695] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:48.434088 31723 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:48.449482 30917 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:48.452190 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:48.452567 527 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20250902 01:58:48.452620 588 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.453779 550 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:48.455741 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 423350us
I20250902 01:58:48.442817 570 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:58:48.469795 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.411155 488 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:48.475754 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 55078.5us
I20250902 01:58:48.436592 30992 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
W20250902 01:58:48.475988 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 55638.1us
I20250902 01:58:48.479205 31001 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:48.490451 30993 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:48.491724 570 raft_consensus.cc:695] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.491935 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:48.492524 30999 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:48.492686 570 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.414527 525 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.497361 31729 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:58:48.498733 31000 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:48.483105 31039 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:48.490464 30964 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.511270 30996 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:48.515394 31730 raft_consensus.cc:1238] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 1. Current term is 2. Ops: []
I20250902 01:58:48.517055 30964 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.449326 522 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250902 01:58:48.492640 30992 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:48.519023 30995 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:48.491730 30993 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250902 01:58:48.519770 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 48454.9us
I20250902 01:58:48.502753 616 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:58:48.432483 426 consensus_queue.cc:237] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:48.526787 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 121359us
W20250902 01:58:48.528667 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 121522us
I20250902 01:58:48.448170 347 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:48.539477 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:48.539935 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.497397 531 consensus_queue.cc:1046] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250902 01:58:48.522394 618 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:48.527704 619 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.547924 30996 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:48.526321 609 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:48.525102 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:48.520172 624 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:48.549739 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.580981 31728 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:48.580371 31695 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.564517 30967 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:48.565209 30964 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.554227 489 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.611766 489 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:48.581488 531 consensus_queue.cc:1230] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer 01f6731ce5044a6a948a5ae04da80e6f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20250902 01:58:48.613736 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49289.2us
I20250902 01:58:48.529249 571 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20250902 01:58:48.539098 616 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:48.528121 32738 raft_consensus.cc:695] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:48.674176 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:48.552384 613 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:48.675333 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:48.682757 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 65413.8us
W20250902 01:58:48.683218 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.675143 31075 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:48.579752 532 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.592011 31071 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250902 01:58:48.697945 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 117644us
I20250902 01:58:48.698931 31695 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:48.580391 508 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:48.672026 609 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:48.548875 622 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.164s
I20250902 01:58:48.609938 525 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:48.581885 614 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.613368 31727 raft_consensus.cc:2374] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2. Re-sending same reply.
I20250902 01:58:48.546171 635 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.170s
I20250902 01:58:48.694545 30892 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:48.701483 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 148149us
I20250902 01:58:48.711582 30967 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.693244 624 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:48.589746 619 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.673341 522 consensus_queue.cc:1046] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20250902 01:58:48.737845 619 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.738080 571 raft_consensus.cc:3053] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:48.738366 619 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.738448 571 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.738900 619 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.739750 625 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.740226 625 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:48.742509 625 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.734110 30964 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.744065 588 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250902 01:58:48.735862 631 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:48.735945 636 raft_consensus.cc:3053] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:48.744339 30964 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:48.754938 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 226188us
I20250902 01:58:48.725265 609 raft_consensus.cc:695] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:58:48.760285 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49209.8us
I20250902 01:58:48.719431 637 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:48.769394 489 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.769459 426 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:48.771188 535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92f16b8a15144fcf8807608bdee71123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:58:48.739233 571 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.772156 571 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:48.773101 30964 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:48.773041 609 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.754354 624 raft_consensus.cc:695] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.774685 575 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:48.775082 575 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.757491 625 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:48.775942 625 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.775453 624 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.776479 575 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.776762 625 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.777552 575 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:48.774688 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.761477 488 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:48.752116 32759 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250902 01:58:48.780331 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:48.807344 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:48.783357 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:48.808959 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.806061 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:48.810581 30917 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:48.754004 636 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.812058 31693 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:48.759395 646 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:48.782264 613 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:48.805759 607 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election won for term 2
I20250902 01:58:48.807049 32759 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:58:48.823118 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.805583 618 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:48.817037 489 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:48.845552 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:48.812070 622 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.019s
I20250902 01:58:48.727015 614 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.761241 32738 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.831272 32759 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:48.852648 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 412082us
W20250902 01:58:48.858093 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 83520.9us
I20250902 01:58:48.858340 30889 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.815819 647 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20250902 01:58:48.842207 489 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:48.844226 607 raft_consensus.cc:695] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:48.870317 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:48.871049 31730 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:48.871452 30992 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250902 01:58:48.860589 31067 consensus_meta.cc:287] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.676s user 0.010s sys 0.003s
I20250902 01:58:48.858757 30964 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.810365 616 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.823254 646 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.855618 570 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:48.871598 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:48.877498 31728 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:48.861066 613 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.861075 517 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250902 01:58:48.872002 347 consensus_queue.cc:1046] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 8, Time since last communication: 0.008s
I20250902 01:58:48.880479 347 consensus_queue.cc:1230] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20250902 01:58:48.880795 30996 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:48.880535 621 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:48.881008 638 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:48.881500 638 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:48.881842 638 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:48.881342 621 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.866909 489 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:48.883664 618 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:48.885002 621 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.879002 30995 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:48.865760 588 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:48.870337 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:48.895272 30964 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:48.860085 426 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250902 01:58:48.897673 634 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.897917 655 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.898223 634 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.898494 655 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:48.900316 634 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.900441 655 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:48.901455 641 log.cc:927] Time spent T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20250902 01:58:48.903174 643 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
W20250902 01:58:48.904610 648 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250902 01:58:48.913995 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:48.886487 624 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250902 01:58:48.903437 482 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.616s user 0.007s sys 0.002s
W20250902 01:58:48.920353 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 68352.9us
W20250902 01:58:48.922194 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:48.916862 588 raft_consensus.cc:695] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:48.915479 30993 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:48.850566 636 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.894526 613 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.208259 614 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:48.967655 650 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.327s
I20250902 01:58:49.295583 347 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.142961 588 consensus_queue.cc:237] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:49.296337 347 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.299418 347 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.287993 30920 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:48.956797 482 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:49.150142 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:49.287835 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 589037us
I20250902 01:58:49.183840 571 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:48.936757 488 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:49.306247 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:49.288123 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 408198us
I20250902 01:58:49.307327 571 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.989876 607 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.919154 624 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:49.308676 652 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:48.968253 628 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.310045 571 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.309819 628 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:48.980188 550 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.331s
I20250902 01:58:49.311762 628 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:48.925509 619 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:48.967936 570 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.341s
W20250902 01:58:49.198704 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.020411 622 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:48.934924 646 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.313244 622 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:49.293638 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 389169us
W20250902 01:58:49.315779 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 401906us
I20250902 01:58:49.308866 624 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:49.303725 31038 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:49.288192 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:49.307554 30964 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:49.319029 31695 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:49.321285 31066 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:49.326539 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.326637 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:49.333990 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.334535 30925 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:48.968761 621 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.317354 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:49.338956 30999 raft_consensus.cc:1238] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:58:49.315402 622 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:49.317896 347 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.304592 625 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:49.313364 613 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.359509 30964 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:49.387457 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.328532 626 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:58:49.389546 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 70560.5us
I20250902 01:58:49.338160 389 consensus_queue.cc:1046] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
W20250902 01:58:49.393648 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.339018 550 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.363895 570 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:49.366425 614 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.364864 532 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:49.404554 30892 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:49.405622 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1821) took 1261 ms. Trace:
I20250902 01:58:49.324575 634 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20250902 01:58:49.387940 31725 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:49.409700 30924 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:49.407954 570 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:49.474434 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.474936 570 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:49.475296 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:49.438851 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 79444.1us
W20250902 01:58:49.477134 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 150485us
I20250902 01:58:49.480774 30964 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:49.365913 621 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.474126 30994 rpcz_store.cc:276] 0902 01:58:48.140617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.140772 (+ 155us) service_pool.cc:225] Handling call
0902 01:58:48.149913 (+ 9141us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:48.280097 (+130184us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.401515 (+1121418us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b5400178754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:49.401705 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138993408}
W20250902 01:58:49.496233 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:49.503420 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.438637 532 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:49.393558 347 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.409786 625 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:49.504948 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.515586 30918 raft_consensus.cc:2391] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:49.470840 613 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:58:49.475965 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45742 (request call id 19) took 14113 ms. Trace:
I20250902 01:58:49.471329 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.439815 626 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:49.402508 618 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20250902 01:58:49.402616 646 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20250902 01:58:49.511004 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.510834 532 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:49.489233 641 log.cc:927] Time spent T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20250902 01:58:49.495442 525 consensus_meta.cc:287] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.769s user 0.008s sys 0.002s
I20250902 01:58:49.493218 607 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.528303 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.517589 482 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.476378 650 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20250902 01:58:49.507849 389 raft_consensus.cc:3053] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Stepping down as leader of term 1
I20250902 01:58:49.438566 614 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:49.517998 508 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.466846 550 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.550518 588 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:49.584388 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:49.586907 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1721) took 1273 ms. Trace:
I20250902 01:58:49.587349 31000 rpcz_store.cc:276] 0902 01:58:48.313681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.313805 (+ 124us) service_pool.cc:225] Handling call
0902 01:58:48.498518 (+184713us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:48.499229 (+ 711us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.586659 (+1087430us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54002ae9d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:49.586840 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424220416}
I20250902 01:58:49.592708 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.555078 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.559830 613 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:49.596000 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.611827 31000 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:49.556998 638 consensus_meta.cc:287] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.675s user 0.007s sys 0.004s
I20250902 01:58:49.579023 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:49.594017 31724 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:49.612835 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 109526us
I20250902 01:58:49.587805 588 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:49.578660 389 raft_consensus.cc:738] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:49.602440 30994 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:49.581583 660 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:49.558072 517 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20250902 01:58:49.579581 607 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.617738 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.901952 31000 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:49.594699 482 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:49.928853 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.009s sys 0.002s
W20250902 01:58:49.929247 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.009s sys 0.002s
W20250902 01:58:49.929522 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 592639us
W20250902 01:58:49.867684 664 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.336s user 0.000s sys 0.000s
W20250902 01:58:49.706993 32759 consensus_meta.cc:287] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.813s user 0.001s sys 0.008s
W20250902 01:58:49.788897 663 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.382s user 0.000s sys 0.001s
W20250902 01:58:49.720665 670 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
W20250902 01:58:49.671276 489 consensus_meta.cc:287] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.788s user 0.008s sys 0.001s
W20250902 01:58:49.789258 666 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.233s user 0.000s sys 0.001s
W20250902 01:58:49.791440 673 log.cc:927] Time spent T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
W20250902 01:58:49.790270 575 consensus_meta.cc:287] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.012s user 0.010s sys 0.000s
W20250902 01:58:49.790825 674 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.239s user 0.000s sys 0.000s
I20250902 01:58:49.616075 665 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:49.936322 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.791451 550 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.793334 637 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:49.954157 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:49.941541 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 536868us
I20250902 01:58:49.793641 634 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.906983 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:49.632771 618 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.899600 660 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.778650 650 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:49.982745 618 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:49.983268 661 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.263s user 0.000s sys 0.000s
I20250902 01:58:49.983031 650 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.985110 618 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.985268 650 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.873005 347 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:49.985924 347 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:49.986341 347 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.987635 698 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:49.525804 31067 rpcz_store.cc:276] 0902 01:58:35.354822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:35.354944 (+ 122us) service_pool.cc:225] Handling call
0902 01:58:35.362718 (+ 7774us) log.cc:844] Serialized 506 byte log entry
0902 01:58:49.405647 (+14042929us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":415,"raft.queue_time_us":8521,"raft.run_cpu_time_us":1538,"raft.run_wall_time_us":14821,"spinlock_wait_cycles":1897856,"thread_start_us":40061,"threads_started":1,"wal-append.queue_time_us":55238,"wal-append.run_cpu_time_us":2345,"wal-append.run_wall_time_us":1071488}
I20250902 01:58:49.948190 30995 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:49.602394 508 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.916954 588 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:49.937666 621 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:49.777278 613 consensus_queue.cc:752] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 8 is ahead of the local log (next sequential op: 8). Destination peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 9, Last known committed idx: 8, Time since last communication: 1.823s
I20250902 01:58:49.997684 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1751) took 1095 ms. Trace:
I20250902 01:58:49.789088 682 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.239s
I20250902 01:58:49.997270 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1615) took 1217 ms. Trace:
I20250902 01:58:49.931284 372 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20250902 01:58:49.998071 30917 rpcz_store.cc:276] 0902 01:58:48.894445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.894800 (+ 355us) service_pool.cc:225] Handling call
0902 01:58:48.897730 (+ 2930us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:48.924225 (+ 26495us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:48.924250 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:48.924669 (+ 419us) log.cc:844] Serialized 26 byte log entry
0902 01:58:49.027116 (+102447us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:49.027171 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.027262 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:49.989948 (+962686us) raft_consensus.cc:1683] finished
0902 01:58:49.989969 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:49.990079 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19005,"raft.queue_time_us":26344,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":102083,"threads_started":1,"wal-append.queue_time_us":464094}
I20250902 01:58:49.643608 638 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:50.006217 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:49.712275 662 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.513s
I20250902 01:58:49.933429 482 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:49.988220 347 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.004211 31075 rpcz_store.cc:276] 0902 01:58:48.772553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.775582 (+ 3029us) service_pool.cc:225] Handling call
0902 01:58:48.775864 (+ 282us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:48.815155 (+ 39291us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:48.815212 (+ 57us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:48.819203 (+ 3991us) log.cc:844] Serialized 26 byte log entry
0902 01:58:49.287382 (+468179us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:49.287419 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.287494 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:49.990014 (+702520us) raft_consensus.cc:1683] finished
0902 01:58:49.990029 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:49.990118 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36232,"raft.queue_time_us":42818,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":3296,"thread_start_us":116596,"threads_started":1,"wal-append.queue_time_us":764999}
I20250902 01:58:49.988293 575 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.012850 684 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:49.982760 660 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:50.013201 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1616) took 1201 ms. Trace:
I20250902 01:58:49.982100 637 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.991991 634 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.945031 32759 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:49.950202 489 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.996722 618 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:50.013640 671 consensus_queue.cc:752] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 8 is ahead of the local log (next sequential op: 8). Destination peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.7, Next index: 9, Last known committed idx: 8, Time since last communication: 1.142s
I20250902 01:58:49.711876 626 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.005239 389 consensus_queue.cc:260] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:49.988279 698 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.008483 650 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:50.003978 621 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.163648 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1755) took 1252 ms. Trace:
W20250902 01:58:50.042433 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:50.261158 31069 rpcz_store.cc:276] 0902 01:58:48.812253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.812400 (+ 147us) service_pool.cc:225] Handling call
0902 01:58:48.820289 (+ 7889us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:48.829221 (+ 8932us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:48.829264 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:48.829759 (+ 495us) log.cc:844] Serialized 26 byte log entry
0902 01:58:48.932786 (+103027us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:48.932827 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:48.932880 (+ 53us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:50.013041 (+1080161us) raft_consensus.cc:1683] finished
0902 01:58:50.013058 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.013156 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7282,"raft.queue_time_us":5939,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":12242,"thread_start_us":78366,"threads_started":1,"wal-append.queue_time_us":495231}
W20250902 01:58:50.077018 31039 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.004s (SENT) [suppressed 42 similar messages]
I20250902 01:58:50.017582 646 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.268939 618 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.014379 680 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.015113 613 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.281790 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 325319us
I20250902 01:58:50.290316 707 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:50.221899 636 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:50.265688 570 consensus_meta.cc:287] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.787s user 0.011s sys 0.000s
I20250902 01:58:50.266986 30922 rpcz_store.cc:276] 0902 01:58:48.913330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.913721 (+ 391us) service_pool.cc:225] Handling call
0902 01:58:48.914073 (+ 352us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:49.313730 (+399657us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:49.313758 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:49.314125 (+ 367us) log.cc:844] Serialized 26 byte log entry
0902 01:58:49.474076 (+159951us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:49.474131 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.474194 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:50.141751 (+667557us) raft_consensus.cc:1683] finished
0902 01:58:50.141774 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.163591 (+ 21817us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399232,"raft.queue_time_us":403376,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7966,"thread_start_us":159618,"threads_started":1,"wal-append.queue_time_us":610667}
I20250902 01:58:50.029489 614 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.027325 625 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.308866 688 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.268913 684 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.309767 646 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.310345 638 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.311067 508 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.260821 482 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.341516 712 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.270983 511 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.344537 712 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.345028 637 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.344916 511 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.311743 618 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.311859 665 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.220618 692 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.259605 694 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.324254 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:50.267730 389 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:50.329064 679 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:50.271102 607 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.215451 687 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:50.332118 709 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.220781 701 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.171373 682 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20250902 01:58:50.306687 662 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.323393 619 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.221875 710 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:50.315423 613 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.272120 575 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.272367 621 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:50.341507 672 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.545s user 0.002s sys 0.000s
I20250902 01:58:50.291273 650 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.351624 634 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.344933 688 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.406492 613 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.408134 688 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.409624 646 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.415342 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1831) took 1520 ms. Trace:
I20250902 01:58:50.333142 707 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.362686 511 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.363618 712 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.364950 698 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.355904 692 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.326810 570 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.330240 671 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.359321 31071 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:50.391633 689 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.395110 662 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.397583 30997 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:50.386956 701 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.398407 710 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.399863 706 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.405603 685 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250902 01:58:50.392807 709 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.387993 30992 raft_consensus.cc:1238] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:50.415830 30996 rpcz_store.cc:276] 0902 01:58:48.895472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.895591 (+ 119us) service_pool.cc:225] Handling call
0902 01:58:48.902862 (+ 7271us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:49.306432 (+403570us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:49.306509 (+ 77us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:49.306907 (+ 398us) log.cc:844] Serialized 26 byte log entry
0902 01:58:49.477556 (+170649us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:49.533130 (+ 55574us) log.cc:844] Serialized 14 byte log entry
0902 01:58:49.578785 (+ 45655us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.579143 (+ 358us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:50.324836 (+745693us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:50.392386 (+ 67550us) raft_consensus.cc:1683] finished
0902 01:58:50.415121 (+ 22735us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.415276 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89335,"prepare.queue_time_us":389636,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":481145,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":86648,"spinlock_wait_cycles":3434240,"thread_start_us":251853,"threads_started":3,"wal-append.queue_time_us":300209}
I20250902 01:58:50.355181 687 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.315533 680 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.439518 30997 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:50.421178 508 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.423414 613 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.434238 511 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.431577 31724 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:50.389369 619 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.455868 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1752) took 1560 ms. Trace:
I20250902 01:58:50.450168 709 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.460409 30997 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.467717 30923 rpcz_store.cc:276] 0902 01:58:48.908691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.909014 (+ 323us) service_pool.cc:225] Handling call
0902 01:58:48.909333 (+ 319us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:49.317329 (+407996us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:49.317368 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:49.324566 (+ 7198us) log.cc:844] Serialized 26 byte log entry
0902 01:58:49.519561 (+194995us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:49.519600 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:49.519667 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:50.455623 (+935956us) raft_consensus.cc:1683] finished
0902 01:58:50.455651 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.455802 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":398234,"raft.queue_time_us":401774,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":109335,"threads_started":1,"wal-append.queue_time_us":800707}
W20250902 01:58:50.459985 624 consensus_meta.cc:287] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.106s user 0.004s sys 0.007s
W20250902 01:58:50.446704 686 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.000s sys 0.014s
I20250902 01:58:50.450379 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:50.387373 694 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.355698 665 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.461194 511 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.441228 671 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.389214 679 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.445399 689 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:50.473580 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.008s sys 0.023s
I20250902 01:58:50.461344 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 792) took 2250 ms. Trace:
I20250902 01:58:50.496889 714 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.497260 684 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:50.498719 532 consensus_meta.cc:287] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.923s user 0.010s sys 0.000s
I20250902 01:58:50.499930 30996 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:50.553625 30997 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 7
W20250902 01:58:50.606384 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.008s sys 0.023s
I20250902 01:58:50.446555 706 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.461365 613 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.557628 30918 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:50.641256 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1591) took 1147 ms. Trace:
W20250902 01:58:50.580127 31725 consensus_meta.cc:287] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 1.061s user 0.006s sys 0.005s
I20250902 01:58:50.651417 30918 rpcz_store.cc:276] 0902 01:58:49.493527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.516165 (+ 22638us) service_pool.cc:225] Handling call
0902 01:58:50.557012 (+1040847us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:50.557236 (+ 224us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.640941 (+ 83705us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.641184 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27594112}
I20250902 01:58:50.467897 508 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.642449 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 315) took 1127 ms. Trace:
I20250902 01:58:50.499117 32759 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.639217 575 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.548326 650 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.612401 708 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.663997 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.000s sys 0.034s
I20250902 01:58:50.592607 619 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.657696 30996 rpcz_store.cc:276] 0902 01:58:49.515370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.459862 (+944492us) service_pool.cc:225] Handling call
0902 01:58:50.499773 (+ 39911us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.642177 (+142404us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.642383 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:50.655192 709 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.650210 639 log.cc:927] Time spent T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.949s user 0.000s sys 0.015s
I20250902 01:58:50.613240 707 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.557516 621 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.682879 710 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.686384 488 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.686944 488 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.611672 624 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.528793 489 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.550806 389 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.634260 525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.042s sys 0.102s
W20250902 01:58:50.698377 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.000s sys 0.022s
W20250902 01:58:50.665966 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:50.654753 31725 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:50.554493 680 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.669744 694 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:50.630738 686 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.008s user 0.000s sys 0.020s
I20250902 01:58:50.669704 30996 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:50.712131 389 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.814026 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 316) took 1287 ms. Trace:
I20250902 01:58:50.814486 30996 rpcz_store.cc:276] 0902 01:58:49.527721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.669020 (+1141299us) service_pool.cc:225] Handling call
0902 01:58:50.669368 (+ 348us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.792724 (+123356us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.813960 (+ 21236us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:50.638801 616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.020s sys 0.051s
I20250902 01:58:50.555649 692 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.663779 30992 raft_consensus.cc:2408] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:58:50.521401 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 788) took 2498 ms. Trace:
I20250902 01:58:50.638108 714 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:50.672140 658 log.cc:927] Time spent T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.516s user 0.003s sys 0.019s
I20250902 01:58:50.676043 662 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.657303 570 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.585358 687 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.586321 701 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.645521 532 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:50.693524 31000 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.016s sys 0.037s
W20250902 01:58:50.889206 31000 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s user 0.016s sys 0.037s
W20250902 01:58:50.693769 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.016s sys 0.037s
W20250902 01:58:50.889775 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.016s sys 0.037s
I20250902 01:58:50.890192 532 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.525030 713 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:50.664330 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.000s sys 0.034s
W20250902 01:58:50.891444 30994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.026s sys 0.124s
I20250902 01:58:50.677731 706 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.797638 389 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.844141 514 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:50.736934 525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s user 0.043s sys 0.103s
W20250902 01:58:50.698143 31067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.011s sys 0.017s
I20250902 01:58:50.681624 709 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:50.702243 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.000s sys 0.022s
I20250902 01:58:50.719916 30997 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:58:50.940317 30995 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
W20250902 01:58:50.878680 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.000s sys 0.020s
I20250902 01:58:50.789227 700 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.883162 671 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.883253 720 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:50.891798 30994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s user 0.027s sys 0.124s
I20250902 01:58:50.788066 30993 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:50.902511 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.956s user 0.032s sys 0.082s
W20250902 01:58:50.952839 30997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1820) took 2801 ms (client timeout 1535 ms). Trace:
W20250902 01:58:50.892124 639 log.cc:927] Time spent T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250902 01:58:50.903306 532 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:50.941845 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.000s sys 0.020s
I20250902 01:58:50.865140 710 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:50.953652 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.007s user 0.032s sys 0.082s
W20250902 01:58:50.800958 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 791) took 2577 ms (client timeout 2004 ms). Trace:
W20250902 01:58:50.954078 30997 rpcz_store.cc:269] 0902 01:58:48.139857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.139966 (+ 109us) service_pool.cc:225] Handling call
0902 01:58:50.397474 (+2257508us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54003e6d20. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020bf550 00007fc602133dc8 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106
0902 01:58:50.941381 (+543907us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":290357504}
I20250902 01:58:50.961155 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:50.963478 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:50.964090 31075 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:50.897279 662 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.966516 31071 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:50.966732 525 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.966650 662 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:50.967509 615 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 7 is higher.
I20250902 01:58:50.967308 525 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.968364 488 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.968734 662 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:50.939867 514 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.969511 525 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.969802 31693 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:50.970163 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1822) took 2553 ms. Trace:
W20250902 01:58:50.963258 31725 rpcz_store.cc:269] 0902 01:58:48.211077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.213499 (+ 2422us) service_pool.cc:225] Handling call
0902 01:58:49.287636 (+1074137us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff19166c 00007fc5ff1915d6 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:50.737254 (+1449618us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5523200}
I20250902 01:58:50.970479 30998 rpcz_store.cc:276] 0902 01:58:48.416746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.416923 (+ 177us) service_pool.cc:225] Handling call
0902 01:58:48.428531 (+ 11608us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:48.432574 (+ 4043us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:48.432602 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:48.437924 (+ 5322us) log.cc:844] Serialized 26 byte log entry
0902 01:58:48.516308 (+ 78384us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:48.521705 (+ 5397us) log.cc:844] Serialized 14 byte log entry
0902 01:58:48.524820 (+ 3115us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:48.525221 (+ 401us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:50.560166 (+2034945us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:50.969919 (+409753us) raft_consensus.cc:1683] finished
0902 01:58:50.969997 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.970100 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31645,"prepare.queue_time_us":5244,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":25575,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":10356,"spinlock_wait_cycles":28359552,"thread_start_us":64082,"threads_started":1,"wal-append.queue_time_us":162823}
I20250902 01:58:50.971911 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:50.972453 31725 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
W20250902 01:58:50.972662 588 consensus_meta.cc:287] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.962s user 0.009s sys 0.000s
I20250902 01:58:50.972394 532 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.973052 588 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.974393 532 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.974866 621 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:50.975348 613 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.975313 621 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.976366 508 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.931751 624 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:50.967012 679 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.977579 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:50.939417 31067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s user 0.011s sys 0.017s
I20250902 01:58:50.975541 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:50.977603 31072 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:50.938956 535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56978 (request call id 358) took 5164 ms. Trace:
I20250902 01:58:50.938555 511 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:50.843607 616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.131s user 0.021s sys 0.051s
I20250902 01:58:50.997234 706 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.963322 694 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:50.907714 30919 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.001s sys 0.014s
I20250902 01:58:51.000317 31066 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:50.962255 700 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:50.891206 713 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:51.123553 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.004s sys 0.014s
I20250902 01:58:51.064368 550 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:58:51.219439 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.888s user 0.007s sys 0.014s
I20250902 01:58:51.016528 620 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:51.220039 550 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:51.126295 660 consensus_meta.cc:287] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.820s user 0.007s sys 0.004s
I20250902 01:58:51.221076 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:51.010962 31039 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.005s sys 0.015s
I20250902 01:58:51.220618 620 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:51.001328 705 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:50.847492 31724 rpcz_store.cc:276] 0902 01:58:48.017165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.017811 (+ 646us) service_pool.cc:225] Handling call
0902 01:58:48.045701 (+ 27890us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:49.517749 (+1472048us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b58004f04b0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020ab003 00007fc60210e577 00007fc602126fa7 00007fc60212879e 00007fc602132af9 00007fc6021333c6 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44
0902 01:58:50.466551 (+948802us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.515531 (+ 48980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89310976}
I20250902 01:58:51.016394 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.077425 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1548) took 2921 ms. Trace:
W20250902 01:58:51.261193 31039 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.991s user 0.005s sys 0.020s
W20250902 01:58:51.280969 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.20404e+06us
I20250902 01:58:51.016296 627 raft_consensus.cc:3053] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
W20250902 01:58:51.279002 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.282393 30892 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:51.054410 689 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:51.284332 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:50.724243 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.007s sys 0.037s
I20250902 01:58:51.259102 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.284778 689 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:51.284984 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.384s user 0.007s sys 0.041s
W20250902 01:58:51.262935 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1746) took 3198 ms (client timeout 2004 ms). Trace:
W20250902 01:58:51.212205 721 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.000s sys 0.018s
I20250902 01:58:50.970594 514 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:51.263940 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 453385us
I20250902 01:58:51.269475 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.064028 665 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:51.212340 709 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:51.168906 588 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:50.608454 31721 rpcz_store.cc:276] 0902 01:58:48.211411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.213393 (+ 1982us) service_pool.cc:225] Handling call
0902 01:58:48.773395 (+560002us) spinlock_profiling.cc:243] Waited 4.35 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff1915ce 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:48.773569 (+ 174us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:48.818437 (+ 44868us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:48.821798 (+ 3361us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:48.823547 (+ 1749us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:48.830471 (+ 6924us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.461103 (+1630632us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b54004d62d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:50.461152 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:50.461273 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98351232}
W20250902 01:58:51.225365 626 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.001s sys 0.012s
W20250902 01:58:51.221727 30919 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s user 0.004s sys 0.018s
W20250902 01:58:51.285292 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.38457e+06us
I20250902 01:58:51.275149 718 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:51.270705 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.228647 706 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:51.285696 30926 rpcz_store.cc:269] 0902 01:58:48.026649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.033013 (+ 6364us) service_pool.cc:225] Handling call
0902 01:58:51.224772 (+3191759us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54002a12d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:51.224873 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":385995392}
I20250902 01:58:51.017431 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1834) took 1481 ms. Trace:
I20250902 01:58:51.291643 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.292848 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:51.290546 609 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.457s user 0.031s sys 0.009s
I20250902 01:58:51.286638 616 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:51.280855 31067 rpcz_store.cc:276] 0902 01:58:48.155982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.156640 (+ 658us) service_pool.cc:225] Handling call
0902 01:58:48.179910 (+ 23270us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:48.180181 (+ 271us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:58:48.886215 (+706034us) log.cc:844] Serialized 14 byte log entry
0902 01:58:50.142211 (+1255996us) log.cc:844] Serialized 14 byte log entry
0902 01:58:50.395891 (+253680us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b1800418ae0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:58:51.008309 (+612418us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:51.038163 (+ 29854us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:58:51.054702 (+ 16539us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.063968 (+ 9266us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:51.077328 (+ 13360us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24460,"prepare.queue_time_us":93835,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":237031,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":43857,"spinlock_wait_cycles":71466240,"thread_start_us":1225309,"threads_started":2,"wal-append.queue_time_us":497182}
I20250902 01:58:51.297971 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.285285 689 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.379225 616 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.386088 689 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:51.386927 616 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:51.387665 31074 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:51.298750 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.299721 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:51.278636 30967 connection.cc:232] RPC call timeout handler was delayed by -1.0700390389999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20250902 01:58:51.389009 30967 connection.cc:232] RPC call timeout handler was delayed by -1.0385390389999429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
W20250902 01:58:51.364635 626 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s user 0.001s sys 0.017s
W20250902 01:58:51.287110 721 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.020s
I20250902 01:58:51.267225 620 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:50.998505 720 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.263063 389 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:51.265614 730 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:51.286263 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.368183 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.094581 532 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:51.370370 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.410470 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:51.293172 621 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:51.293332 687 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:51.259402 694 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.303694 627 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:51.377045 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.267088 660 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:50.692286 708 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.382433 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:50.991700 710 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.390473 709 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:51.393186 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:51.393492 30967 leader_election.cc:336] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.548s (SENT)
W20250902 01:58:51.380002 609 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.545s user 0.031s sys 0.009s
I20250902 01:58:51.396097 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:51.439633 686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.026s sys 0.056s
W20250902 01:58:51.453032 686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.026s sys 0.056s
I20250902 01:58:51.454277 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1733) took 2998 ms. Trace:
I20250902 01:58:51.455992 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.456750 31725 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:51.457056 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.458149 31721 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:51.408190 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.459767 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:51.460140 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.460492 31067 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:51.461405 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:51.459815 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.463311 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.463946 31729 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 1.
I20250902 01:58:51.449553 709 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.464876 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.465548 31729 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:51.409556 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.411315 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.414984 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.415182 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.468097 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.468218 31071 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:51.407673 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.459271 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.469861 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:51.401700 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 392) took 1447 ms. Trace:
I20250902 01:58:51.470698 30919 rpcz_store.cc:276] 0902 01:58:49.953992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.959682 (+ 5690us) service_pool.cc:225] Handling call
0902 01:58:50.092740 (+133058us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.152593 (+ 59853us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:58:50.312164 (+159571us) log.cc:844] Serialized 14 byte log entry
0902 01:58:51.374147 (+1061983us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:51.378145 (+ 3998us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:51.383450 (+ 5305us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.391153 (+ 7703us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:51.399829 (+ 8676us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":687619,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":16822,"raft.queue_time_us":5268,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":5451,"thread_start_us":1141486,"threads_started":2,"wal-append.queue_time_us":953548}
I20250902 01:58:51.473006 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1735) took 2075 ms. Trace:
I20250902 01:58:51.473172 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.473937 30919 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:51.378311 718 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.420764 31074 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:51.469900 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1739) took 2069 ms. Trace:
I20250902 01:58:51.477876 31000 rpcz_store.cc:276] 0902 01:58:49.400770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.921825 (+521055us) service_pool.cc:225] Handling call
0902 01:58:49.923094 (+ 1269us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:49.989128 (+ 66034us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:49.989387 (+ 259us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:50.011340 (+ 21953us) log.cc:844] Serialized 26 byte log entry
0902 01:58:50.889533 (+878193us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:50.889575 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.889627 (+ 52us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:51.469674 (+580047us) raft_consensus.cc:1683] finished
0902 01:58:51.469698 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:51.469836 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":226513,"spinlock_wait_cycles":12017024,"thread_start_us":805677,"threads_started":2,"wal-append.queue_time_us":1398968}
I20250902 01:58:51.428668 31001 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250902 01:58:51.446218 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 56100us
I20250902 01:58:51.448161 30997 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:51.454747 30999 rpcz_store.cc:276] 0902 01:58:48.455460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.493064 (+ 37604us) service_pool.cc:225] Handling call
0902 01:58:48.493340 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:49.364076 (+870736us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.453970 (+2089894us) spinlock_profiling.cc:243] Waited 845 ms on lock 0x7b54002bdc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:51.454193 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944628096}
I20250902 01:58:51.457944 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.473510 30993 rpcz_store.cc:276] 0902 01:58:49.397822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.401242 (+ 3420us) service_pool.cc:225] Handling call
0902 01:58:49.495045 (+ 93803us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.953575 (+1458530us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.472730 (+519155us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54002bd754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:51.472924 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2966433152}
I20250902 01:58:51.469224 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.492959 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:51.497331 30922 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:51.494624 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1730) took 3039 ms. Trace:
I20250902 01:58:51.495218 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.499311 30918 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:51.495837 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 48843.2us
I20250902 01:58:51.282852 705 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.492368 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.518198 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.501730 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.504712 30892 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:51.505465 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.506644 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:51.520514 31000 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 5
I20250902 01:58:51.497933 31001 rpcz_store.cc:276] 0902 01:58:48.454716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.493069 (+ 38353us) service_pool.cc:225] Handling call
0902 01:58:51.412739 (+2919670us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:51.412953 (+ 214us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:51.480872 (+ 67919us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.494560 (+ 13688us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6891904}
W20250902 01:58:51.438863 30889 connection.cc:232] RPC call timeout handler was delayed by -1.9013606659999596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
W20250902 01:58:51.524549 31692 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.045s sys 0.103s
W20250902 01:58:51.659109 727 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
W20250902 01:58:51.503099 722 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.335s user 0.000s sys 0.001s
I20250902 01:58:51.429180 687 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:51.718107 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:51.718988 30926 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:51.518992 655 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:51.762813 30922 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
W20250902 01:58:51.727785 31692 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.837s user 0.050s sys 0.124s
I20250902 01:58:51.738989 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:51.450688 347 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.210s
W20250902 01:58:51.763612 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 873540us
W20250902 01:58:51.764261 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.764164 30993 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
W20250902 01:58:51.764952 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.373075 30992 rpcz_store.cc:276] 0902 01:58:49.536402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.960095 (+1423693us) service_pool.cc:225] Handling call
0902 01:58:51.016884 (+ 56789us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:51.017089 (+ 205us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:51.017118 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:51.017162 (+ 44us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:51.017212 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.017274 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:51.017374 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:51.659360 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.518100 632 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:51.765377 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.412072 389 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:51.439919 532 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:51.412254 730 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.769941 627 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:51.967826 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:51.799343 31000 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:51.847507 30997 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250902 01:58:52.030223 31728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.007s sys 0.017s
W20250902 01:58:52.381649 31728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.966s user 0.007s sys 0.037s
I20250902 01:58:52.382769 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:52.383455 31000 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:51.904453 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.052s sys 0.098s
W20250902 01:58:52.385504 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.394s user 0.062s sys 0.115s
I20250902 01:58:52.386773 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.392427 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:52.333510 627 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:52.385866 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.41618e+06us
I20250902 01:58:51.924908 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:51.897552 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:51.817466 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:51.783897 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:51.835968 30918 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:51.019135 535 rpcz_store.cc:276] 0902 01:58:45.789517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:46.339460 (+549943us) service_pool.cc:225] Handling call
0902 01:58:46.525634 (+186174us) ts_tablet_manager.cc:928] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
0902 01:58:48.548405 (+2022771us) tablet_copy_client.cc:368] Tablet copy session begun
0902 01:58:50.469762 (+1921357us) tablet_metadata.cc:641] Metadata flushed
0902 01:58:50.529535 (+ 59773us) tablet_copy_client.cc:497] Wrote new tablet metadata
0902 01:58:50.887024 (+357489us) tablet_copy_client.cc:418] Wrote new consensus metadata
0902 01:58:50.887344 (+ 320us) tablet_replica.cc:206] Creating consensus instance
0902 01:58:50.903177 (+ 15833us) ts_tablet_manager.cc:983] Replica registered.
0902 01:58:50.938858 (+ 35681us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":4568,"tablet-copy.queue_time_us":142992,"thread_start_us":124299,"threads_started":2}
I20250902 01:58:52.396373 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:52.420202 30917 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:52.420396 30997 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:52.420995 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1767) took 1048 ms. Trace:
I20250902 01:58:52.421458 30917 rpcz_store.cc:276] 0902 01:58:51.372759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.761186 (+388427us) service_pool.cc:225] Handling call
0902 01:58:52.419913 (+658727us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.420679 (+ 766us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.420925 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:52.396266 31075 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.032s user 0.006s sys 0.023s
W20250902 01:58:52.405354 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 732503us
I20250902 01:58:51.897694 616 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:51.986058 733 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.620s user 0.001s sys 0.000s
I20250902 01:58:52.400528 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.379802 694 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:51.797508 655 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:52.416067 708 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.194597 710 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:52.418587 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.006s sys 0.009s
I20250902 01:58:51.779728 682 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:52.453678 682 raft_consensus.cc:695] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:52.454599 682 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:52.455971 745 log.cc:927] Time spent T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.394s user 0.000s sys 0.001s
W20250902 01:58:52.235074 620 consensus_meta.cc:287] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.826s user 0.005s sys 0.002s
I20250902 01:58:52.457075 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 803) took 1489 ms. Trace:
I20250902 01:58:52.456627 620 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:52.457463 742 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.442s user 0.001s sys 0.000s
I20250902 01:58:52.458828 31730 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:52.418851 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.459419 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 806) took 1490 ms. Trace:
I20250902 01:58:52.455313 30997 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:52.459251 620 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:52.457504 31727 rpcz_store.cc:276] 0902 01:58:50.967768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.969272 (+ 1504us) service_pool.cc:225] Handling call
0902 01:58:50.969605 (+ 333us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:50.974543 (+ 4938us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:50.974570 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:51.002160 (+ 27590us) log.cc:844] Serialized 26 byte log entry
0902 01:58:51.421087 (+418927us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:51.421127 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.421188 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:52.456811 (+1035623us) raft_consensus.cc:1683] finished
0902 01:58:52.456864 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.456996 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4541,"raft.queue_time_us":6986,"spinlock_wait_cycles":7486720,"thread_start_us":393116,"threads_started":1,"wal-append.queue_time_us":946081}
I20250902 01:58:52.461920 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.438123 347 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:52.464068 347 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:52.466022 347 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:52.466511 347 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:52.466979 347 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:52.071530 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.397811 30999 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:52.478148 31066 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
W20250902 01:58:52.422886 31075 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.059s user 0.006s sys 0.023s
I20250902 01:58:52.401381 31001 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
W20250902 01:58:52.397241 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.002s sys 0.016s
W20250902 01:58:52.480494 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.003s sys 0.017s
W20250902 01:58:52.480749 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 740513us
I20250902 01:58:52.481391 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:52.400132 31000 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:58:52.032039 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.355774 709 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:52.430837 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:52.398543 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:52.172343 31072 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.002s sys 0.029s
I20250902 01:58:52.432174 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:52.527645 31072 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.003s sys 0.037s
I20250902 01:58:51.877449 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.528414 30993 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:52.528601 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1558) took 1099 ms. Trace:
I20250902 01:58:51.926867 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1738) took 2526 ms. Trace:
I20250902 01:58:51.969897 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:52.042719 31722 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.015s sys 0.015s
W20250902 01:58:52.531033 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 49680.4us
W20250902 01:58:52.401507 30996 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.005s sys 0.024s
I20250902 01:58:51.777062 30921 raft_consensus.cc:2374] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4. Re-sending same reply.
I20250902 01:58:52.211627 660 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.439299 31073 raft_consensus.cc:2408] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250902 01:58:52.432715 30998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.966s user 0.000s sys 0.012s
I20250902 01:58:51.762523 687 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:52.436084 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 414529us
I20250902 01:58:52.442431 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.445416 655 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:52.445842 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.006s sys 0.009s
W20250902 01:58:52.541045 30998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.074s user 0.000s sys 0.012s
W20250902 01:58:52.395771 726 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.007s sys 0.017s
I20250902 01:58:52.450179 389 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.438792 616 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.441370 711 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:52.418243 705 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:52.464076 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.459831 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:52.433944 721 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.003s sys 0.007s
I20250902 01:58:51.969834 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 311) took 2455 ms. Trace:
I20250902 01:58:51.825925 30919 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:52.072926 706 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:51.976751 31724 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:51.846565 701 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:52.402276 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1761) took 1035 ms. Trace:
W20250902 01:58:52.429658 30925 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.018s sys 0.069s
I20250902 01:58:52.459786 31723 rpcz_store.cc:276] 0902 01:58:50.968717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.975635 (+ 6918us) service_pool.cc:225] Handling call
0902 01:58:50.975949 (+ 314us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:58:50.976302 (+ 353us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:50.976332 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:58:50.997201 (+ 20869us) log.cc:844] Serialized 52 byte log entry
0902 01:58:51.409342 (+412141us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:51.414948 (+ 5606us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.415019 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:52.459221 (+1044202us) raft_consensus.cc:1683] finished
0902 01:58:52.459240 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.459365 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":858,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10728,"thread_start_us":404428,"threads_started":1,"wal-append.queue_time_us":729821}
I20250902 01:58:52.487628 31068 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:52.491302 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 802) took 1531 ms. Trace:
I20250902 01:58:52.491852 347 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:51.847842 632 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.505270 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1621) took 2189 ms. Trace:
I20250902 01:58:52.507493 31726 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:52.441325 535 tablet_copy_client.cc:806] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:58:52.520593 730 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:51.890548 713 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:52.532153 30996 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s user 0.006s sys 0.024s
I20250902 01:58:52.052484 700 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:52.536494 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:52.535266 31722 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.267s user 0.020s sys 0.020s
I20250902 01:58:52.529343 30994 rpcz_store.cc:276] 0902 01:58:49.400517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.921403 (+520886us) service_pool.cc:225] Handling call
0902 01:58:49.921712 (+ 309us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:49.922012 (+ 300us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:49.922047 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:49.931358 (+ 9311us) log.cc:844] Serialized 26 byte log entry
0902 01:58:50.953372 (+1022014us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:50.953410 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:50.953470 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:51.926606 (+973136us) raft_consensus.cc:1683] finished
0902 01:58:51.926633 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:51.926798 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":212587,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":23727,"thread_start_us":974115,"threads_started":1,"wal-append.queue_time_us":1311778}
I20250902 01:58:52.538115 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:52.477077 31067 consensus_meta.cc:287] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.016s user 0.008s sys 0.001s
I20250902 01:58:52.437336 718 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.543999 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.808614 30926 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:52.529098 31072 rpcz_store.cc:276] 0902 01:58:51.429616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.467198 (+ 37582us) service_pool.cc:225] Handling call
0902 01:58:51.492031 (+ 24833us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.528145 (+1036114us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:52.528187 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:52.528239 (+ 52us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:52.528290 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.528369 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.528541 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6,"raft.queue_time_us":772,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":121,"thread_start_us":1026457,"threads_started":1}
I20250902 01:58:52.482954 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.550963 616 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.563583 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:52.799281 30925 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.381s user 0.023s sys 0.074s
W20250902 01:58:52.799842 31692 connection.cc:232] RPC call timeout handler was delayed by -1.5672470470000235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
I20250902 01:58:52.563802 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1753) took 3653 ms. Trace:
W20250902 01:58:52.817818 31692 connection.cc:232] RPC call timeout handler was delayed by -1.2317726590000575s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25720000000000004s for a response
W20250902 01:58:52.818378 31692 connection.cc:232] RPC call timeout handler was delayed by -1.1650988110000071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
I20250902 01:58:52.817951 30924 rpcz_store.cc:276] 0902 01:58:48.909954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.910388 (+ 434us) service_pool.cc:225] Handling call
0902 01:58:48.910661 (+ 273us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:49.475741 (+565080us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.563547 (+3087806us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b5400240ed4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:52.563711 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1147345664}
W20250902 01:58:52.818979 31692 connection.cc:232] RPC call timeout handler was delayed by -1.0500175710000121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27540000000000003s for a response
W20250902 01:58:52.635193 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 52770.7us
I20250902 01:58:52.542500 627 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:58:52.717041 31038 connection.cc:232] RPC call timeout handler was delayed by -2.9013977160000195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20250902 01:58:52.553627 30964 connection.cc:232] RPC call timeout handler was delayed by -1.1779002129999299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
I20250902 01:58:52.798847 31069 rpcz_store.cc:276] 0902 01:58:50.316293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.316757 (+ 464us) service_pool.cc:225] Handling call
0902 01:58:50.317043 (+ 286us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.635112 (+318069us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:50.635144 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:50.635180 (+ 36us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:50.635217 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.504979 (+1869762us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54000f3cd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:52.505059 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.505206 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731127040}
I20250902 01:58:52.632737 660 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:52.801539 30995 rpcz_store.cc:276] 0902 01:58:49.514327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.997539 (+483212us) service_pool.cc:225] Handling call
0902 01:58:50.940112 (+942573us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:50.940813 (+ 701us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:51.969619 (+1028806us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b54000b5c50. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:51.969772 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2130108288}
W20250902 01:58:52.546036 726 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.007s sys 0.018s
I20250902 01:58:52.800925 31728 rpcz_store.cc:276] 0902 01:58:50.960385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.960513 (+ 128us) service_pool.cc:225] Handling call
0902 01:58:50.970517 (+ 10004us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:51.397682 (+427165us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:51.397717 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:51.402117 (+ 4400us) log.cc:844] Serialized 26 byte log entry
0902 01:58:52.382136 (+980019us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:52.382191 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.382281 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:52.491094 (+108813us) raft_consensus.cc:1683] finished
0902 01:58:52.491111 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.491252 (+ 141us) inbound_call.cc:173] Queueing success response
0902 01:58:52.506222 (+ 14970us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b48002709d8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
Metrics: {"mutex_wait_us":27027,"raft.queue_time_us":74242,"raft.run_cpu_time_us":12432,"raft.run_wall_time_us":1451188,"spinlock_wait_cycles":17476480,"thread_start_us":1355662,"threads_started":2,"wal-append.queue_time_us":1070608}
W20250902 01:58:52.816902 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.066s sys 0.139s
W20250902 01:58:52.829524 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.927s user 0.067s sys 0.139s
W20250902 01:58:52.801304 721 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.387s user 0.007s sys 0.020s
I20250902 01:58:52.829807 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:52.823899 31692 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:52.830547 31729 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:52.799599 30918 rpcz_store.cc:276] 0902 01:58:51.367012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.530772 (+163760us) service_pool.cc:225] Handling call
0902 01:58:51.782828 (+252056us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.401978 (+619150us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.402205 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:52.825735 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 393) took 2866 ms. Trace:
W20250902 01:58:52.827600 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:52.827469 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.829802 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.547063 31721 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:52.832063 31728 raft_consensus.cc:2391] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 2: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:52.832904 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.834065 30924 raft_consensus.cc:2408] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250902 01:58:52.835070 30964 connection.cc:232] RPC call timeout handler was delayed by -1.1590002129998993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
I20250902 01:58:52.827163 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 807) took 1856 ms. Trace:
I20250902 01:58:52.836201 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:52.837780 31038 connection.cc:232] RPC call timeout handler was delayed by -1.6831440770000654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20250902 01:58:52.838258 31038 connection.cc:232] RPC call timeout handler was delayed by -1.3618440770000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
W20250902 01:58:52.838634 31038 connection.cc:232] RPC call timeout handler was delayed by -1.3618440770000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
W20250902 01:58:52.838966 31038 connection.cc:232] RPC call timeout handler was delayed by -1.2277440770000112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
I20250902 01:58:52.839649 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:52.806244 31067 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
W20250902 01:58:52.846750 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:52.847560 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 452815us
I20250902 01:58:52.811686 31727 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:52.581573 711 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.831230 30925 rpcz_store.cc:276] 0902 01:58:49.959594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:49.993483 (+ 33889us) service_pool.cc:225] Handling call
0902 01:58:51.222009 (+1228526us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:51.222197 (+ 188us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:51.258815 (+ 36618us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:51.265223 (+ 6408us) log.cc:844] Serialized 14 byte log entry
0902 01:58:51.418672 (+153449us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b1800449440. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d742da 00007fc60213a9da
0902 01:58:52.825432 (+1406760us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:52.825483 (+ 51us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:52.825542 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.825596 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.825685 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6203,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":4046,"raft.queue_time_us":6569,"spinlock_wait_cycles":51526656,"thread_start_us":1398894,"threads_started":1,"wal-append.queue_time_us":1199051}
I20250902 01:58:52.812915 511 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:52.847710 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.860262 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.861845 30917 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:52.827016 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 539) took 1400 ms. Trace:
I20250902 01:58:52.821765 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1763) took 1442 ms. Trace:
I20250902 01:58:52.863377 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:52.848894 30964 connection.cc:232] RPC call timeout handler was delayed by -1.0321002129999215s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
I20250902 01:58:52.825109 627 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:52.808845 701 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.864111 30924 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:52.865228 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.869654 30926 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:52.870949 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:52.864280 30964 connection.cc:232] RPC call timeout handler was delayed by -1.0285002129999157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20250902 01:58:52.837042 347 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:52.871488 31073 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:52.828589 535 tablet_copy_client.cc:670] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:58:52.876485 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.876660 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.876757 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.880072 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.884784 31000 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:52.846966 743 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:52.853217 616 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:52.863595 758 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:52.881048 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.888152 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.860306 647 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:52.867592 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1631) took 1456 ms. Trace:
I20250902 01:58:52.885586 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.906389 31000 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:52.887969 759 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20250902 01:58:52.886576 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.850546 700 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:52.849625 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 388) took 1011 ms. Trace:
I20250902 01:58:52.865325 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 398) took 1021 ms. Trace:
I20250902 01:58:52.871137 627 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:52.837616 660 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.855059 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.888226 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.888345 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.889595 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.890450 711 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.872133 532 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:52.882151 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.888556 743 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:52.902235 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.903443 31729 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:52.908170 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1754) took 3997 ms. Trace:
I20250902 01:58:52.936429 30920 rpcz_store.cc:276] 0902 01:58:48.911066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.911176 (+ 110us) service_pool.cc:225] Handling call
0902 01:58:49.287817 (+376641us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:49.300271 (+ 12454us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.905012 (+3604741us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b540034f8d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:52.908097 (+ 3085us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77908736}
I20250902 01:58:52.937630 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.938344 30920 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:52.939077 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:52.908031 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.939779 743 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:52.836287 31722 rpcz_store.cc:276] 0902 01:58:50.969037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.970711 (+ 1674us) service_pool.cc:225] Handling call
0902 01:58:50.970987 (+ 276us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:50.997366 (+ 26379us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:50.997398 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:51.016146 (+ 18748us) log.cc:844] Serialized 26 byte log entry
0902 01:58:52.810361 (+1794215us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:52.812226 (+ 1865us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.812314 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:52.812372 (+ 58us) raft_consensus.cc:1683] finished
0902 01:58:52.812387 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.825305 (+ 12918us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56924,"raft.queue_time_us":69662,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":10002,"thread_start_us":1534647,"threads_started":1,"wal-append.queue_time_us":1275260}
I20250902 01:58:52.940280 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.940712 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:52.941231 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.941769 31066 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:52.942546 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.943074 31066 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:52.906116 31067 rpcz_store.cc:276] 0902 01:58:51.404415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.459470 (+ 55055us) service_pool.cc:225] Handling call
0902 01:58:52.854745 (+1395275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:52.944067 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.944582 705 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:58:52.945011 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.910297 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.946084 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.947081 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:52.910833 514 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:52.912165 30926 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:52.949515 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.950733 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.918018 31727 rpcz_store.cc:276] 0902 01:58:51.841406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.462756 (+621350us) service_pool.cc:225] Handling call
0902 01:58:52.862981 (+400225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:52.885452 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.875483 635 consensus_queue.cc:1046] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250902 01:58:52.951866 635 consensus_queue.cc:1230] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer 975ecd397f5146409baac87e56a302e9 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250902 01:58:52.917755 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.952306 754 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:52.952474 660 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:52.873659 706 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250902 01:58:52.952730 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 391) took 1113 ms. Trace:
I20250902 01:58:52.953075 30994 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:58:52.866890 30919 rpcz_store.cc:276] 0902 01:58:51.371313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.474758 (+103445us) service_pool.cc:225] Handling call
0902 01:58:52.809938 (+1335180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:52.953220 31727 rpcz_store.cc:276] 0902 01:58:51.839506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.951792 (+1112286us) service_pool.cc:225] Handling call
0902 01:58:52.952083 (+ 291us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.952355 (+ 272us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:52.952401 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:52.952466 (+ 65us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:52.952505 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.952570 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.952681 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1420,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88}
I20250902 01:58:52.921178 700 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.955052 31727 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:52.905356 616 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:52.956250 751 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250902 01:58:52.957696 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1837) took 1510 ms. Trace:
I20250902 01:58:52.888741 347 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:52.877163 556 consensus_queue.cc:1046] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20250902 01:58:52.922900 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51869.3us
I20250902 01:58:52.894057 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.981781 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.982702 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.983090 556 raft_consensus.cc:3053] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:52.924276 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 53529.3us
I20250902 01:58:52.983485 556 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:52.901199 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.927472 30924 raft_consensus.cc:2391] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:52.863036 31724 rpcz_store.cc:276] 0902 01:58:51.423968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.486197 (+ 62229us) service_pool.cc:225] Handling call
0902 01:58:51.783848 (+297651us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.800132 (+1016284us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.824094 (+ 23962us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:52.925280 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:52.910836 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1624) took 1644 ms. Trace:
I20250902 01:58:52.961896 31074 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:52.904124 725 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.949s user 0.006s sys 0.011s
I20250902 01:58:52.962777 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 392) took 1123 ms. Trace:
I20250902 01:58:52.963799 31729 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:52.962608 30920 raft_consensus.cc:2391] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:52.965576 31726 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:52.962721 30892 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:52.941256 31722 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:52.967800 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 390) took 1128 ms. Trace:
I20250902 01:58:52.957713 720 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:52.968890 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:52.952667 754 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:52.948256 514 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:52.973044 627 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.958163 30998 rpcz_store.cc:276] 0902 01:58:51.447660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.447871 (+ 211us) service_pool.cc:225] Handling call
0902 01:58:51.465591 (+ 17720us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:58:51.466016 (+ 425us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:51.466055 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:58:51.466707 (+ 652us) log.cc:844] Serialized 52 byte log entry
0902 01:58:52.541364 (+1074657us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:52.541397 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.541449 (+ 52us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:52.957401 (+415952us) raft_consensus.cc:1683] finished
0902 01:58:52.957434 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.957626 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15642,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":29447,"thread_start_us":1074231,"threads_started":1,"wal-append.queue_time_us":1365196}
I20250902 01:58:52.975485 669 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:52.976608 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:52.977376 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 233606us
I20250902 01:58:52.878122 511 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.974440 687 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.951956 30918 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:52.947362 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.984324 556 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:52.905550 758 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:52.996829 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.009s sys 0.005s
W20250902 01:58:53.549132 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s user 0.040s sys 0.043s
I20250902 01:58:53.556468 693 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:53.453759 30999 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.036s sys 0.037s
W20250902 01:58:53.558674 30999 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.643s user 0.038s sys 0.043s
I20250902 01:58:53.559206 31727 rpcz_store.cc:276] 0902 01:58:51.839847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.954532 (+1114685us) service_pool.cc:225] Handling call
0902 01:58:52.954796 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.955427 (+ 631us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.962732 (+ 7305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.560638 30917 raft_consensus.cc:2408] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250902 01:58:53.561108 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1596) took 1019 ms. Trace:
I20250902 01:58:53.561489 30917 rpcz_store.cc:276] 0902 01:58:52.541461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.926288 (+384827us) service_pool.cc:225] Handling call
0902 01:58:53.561056 (+634768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.165906 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.279135 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.561270 30926 raft_consensus.cc:2408] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:53.563727 682 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:53.563834 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:53.564447 30998 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 1 FOLLOWER]: Advancing to term 2
I20250902 01:58:53.564410 682 raft_consensus.cc:695] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:53.565784 682 consensus_queue.cc:237] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:53.568280 627 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:53.569715 725 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.634s user 0.017s sys 0.011s
I20250902 01:58:53.067350 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.095280 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.571694 31001 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:53.572564 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.573055 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.573278 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1604) took 1029 ms. Trace:
W20250902 01:58:53.066736 30965 leader_election.cc:336] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.583s (SENT)
I20250902 01:58:53.611917 30924 rpcz_store.cc:276] 0902 01:58:52.543364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.572329 (+1028965us) service_pool.cc:225] Handling call
0902 01:58:53.573229 (+ 900us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:53.612110 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 545441us
I20250902 01:58:53.613294 31001 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:58:53.318929 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:53.574159 31725 rpcz_store.cc:276] 0902 01:58:51.839250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.808823 (+969573us) service_pool.cc:225] Handling call
0902 01:58:52.967735 (+158912us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.603108 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.121659 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.549830 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.639436 30920 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:53.549307 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 405) took 1113 ms. Trace:
I20250902 01:58:53.647815 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 799) took 5159 ms. Trace:
I20250902 01:58:53.579277 30967 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:53.549767 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:53.592633 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:53.550107 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:53.597393 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.001s sys 0.010s
W20250902 01:58:53.016404 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:53.599611 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 636977us
I20250902 01:58:53.574374 31729 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:53.571280 31074 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:53.562836 30993 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:53.575904 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1849) took 1428 ms. Trace:
I20250902 01:58:52.917213 31721 rpcz_store.cc:276] 0902 01:58:51.838627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.846202 (+ 7575us) service_pool.cc:225] Handling call
0902 01:58:52.847300 (+1001098us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.548767 31075 rpcz_store.cc:276] 0902 01:58:51.266094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.266360 (+ 266us) service_pool.cc:225] Handling call
0902 01:58:51.289722 (+ 23362us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:51.289936 (+ 214us) raft_consensus.cc:1532] Early marking committed up to index 4
0902 01:58:52.478818 (+1188882us) log.cc:844] Serialized 14 byte log entry
0902 01:58:52.910550 (+431732us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:52.910594 (+ 44us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:52.910629 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:52.910688 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:52.910791 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5993,"prepare.queue_time_us":1152938,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":2580,"raft.queue_time_us":2861,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":5046,"thread_start_us":1543119,"threads_started":2,"wal-append.queue_time_us":516468}
I20250902 01:58:53.648006 31069 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:53.656953 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.657562 30993 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:53.650476 31692 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:53.649132 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 69919.8us
W20250902 01:58:53.658668 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 108641us
W20250902 01:58:53.652595 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:53.350325 764 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
I20250902 01:58:53.657562 31727 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:58:53.663331 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:52.978502 660 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:53.650694 31066 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
W20250902 01:58:53.650987 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.002s sys 0.011s
I20250902 01:58:53.671288 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:53.351001 752 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20250902 01:58:53.662936 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 394) took 1811 ms. Trace:
I20250902 01:58:53.664052 30924 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:53.674731 31729 rpcz_store.cc:276] 0902 01:58:51.840379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.963332 (+1122953us) service_pool.cc:225] Handling call
0902 01:58:52.963670 (+ 338us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.651660 (+687990us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.651816 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.663411 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.676023 31074 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:53.664570 30889 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:53.648445 31726 rpcz_store.cc:276] 0902 01:58:48.427615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.427725 (+ 110us) service_pool.cc:225] Handling call
0902 01:58:48.433809 (+ 6084us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.506965 (+4073156us) spinlock_profiling.cc:243] Waited 90.7 ms on lock 0x7b58005411b0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020ab003 00007fc60210e577 00007fc602126fa7 00007fc60212879e 00007fc602132af9 00007fc6021333c6 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44
0902 01:58:53.556270 (+1049305us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.587214 (+ 30944us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208540032}
W20250902 01:58:53.671785 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 750572us
I20250902 01:58:53.664088 30923 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:53.351572 31724 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:53.672093 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:53.673877 30892 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:53.675310 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1602) took 1132 ms. Trace:
I20250902 01:58:53.677439 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:53.677472 30924 rpcz_store.cc:276] 0902 01:58:52.542842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.648236 (+1105394us) service_pool.cc:225] Handling call
0902 01:58:53.648497 (+ 261us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.674990 (+ 26493us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.675244 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:53.597772 662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.003s sys 0.011s
I20250902 01:58:53.670158 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.681049 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:53.681782 30995 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:53.676226 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.352509 701 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.684015 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.684131 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.684594 31069 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:53.678807 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.685616 31692 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:53.685806 31726 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:53.679071 700 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.679374 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.682580 30920 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:53.680500 30918 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:53.688042 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:53.686333 30924 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:53.687510 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1606) took 1140 ms. Trace:
I20250902 01:58:53.689556 30889 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:53.689932 30920 rpcz_store.cc:276] 0902 01:58:52.546760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.649238 (+1102478us) service_pool.cc:225] Handling call
0902 01:58:53.675195 (+ 25957us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.687286 (+ 12091us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.687465 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.686844 31075 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:53.557018 693 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.686343 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 400) took 1603 ms. Trace:
W20250902 01:58:53.674321 763 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.511s user 0.001s sys 0.000s
W20250902 01:58:53.700402 475 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20250902 01:58:53.648006 31722 rpcz_store.cc:276] 0902 01:58:52.435799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.940458 (+504659us) service_pool.cc:225] Handling call
0902 01:58:53.549254 (+608796us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.701180 31729 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:53.680188 660 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.670413 761 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:53.657596 30992 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:58:53.736478 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 96671.7us
I20250902 01:58:53.687348 682 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:53.699672 701 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:53.701114 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.670089 721 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:53.738857 721 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:53.739215 721 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:53.739694 31073 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:53.679095 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.663319 720 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:53.688526 30967 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:53.701854 31726 rpcz_store.cc:276] 0902 01:58:52.082677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.685307 (+1602630us) service_pool.cc:225] Handling call
0902 01:58:53.685539 (+ 232us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.686136 (+ 597us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.686292 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.702937 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.705133 31727 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
W20250902 01:58:53.743922 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 55460.7us
I20250902 01:58:53.744130 31066 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:53.650274 669 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:53.673955 532 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:53.699697 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 399) took 1858 ms. Trace:
I20250902 01:58:53.708532 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.691844 730 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:53.708603 754 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.690444 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:53.680258 662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.003s sys 0.011s
I20250902 01:58:53.712648 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.696388 31075 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:58:53.708110 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1844) took 1832 ms. Trace:
I20250902 01:58:53.770813 30997 rpcz_store.cc:276] 0902 01:58:51.875391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.460291 (+584900us) service_pool.cc:225] Handling call
0902 01:58:53.707215 (+1246924us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:58:53.707439 (+ 224us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.707684 (+ 245us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:53.707721 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:53.707779 (+ 58us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:53.707824 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.707921 (+ 97us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:53.708038 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42369280}
I20250902 01:58:53.692185 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.713938 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.686331 757 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:53.715147 31067 raft_consensus.cc:1238] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:53.716507 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.783452 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.784018 31074 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:53.716065 31069 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:53.715448 517 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20250902 01:58:53.712574 655 consensus_meta.cc:287] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.164s user 0.004s sys 0.005s
I20250902 01:58:53.733013 31722 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:58:53.692019 775 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:53.743530 31001 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:53.701786 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 406) took 1255 ms. Trace:
I20250902 01:58:53.734123 347 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:53.755851 31725 raft_consensus.cc:2408] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:58:53.756083 758 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.741509 706 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250902 01:58:53.795109 30922 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1769) took 2421 ms (client timeout 3085 ms). Trace:
W20250902 01:58:53.758806 31039 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.004s (SENT)
I20250902 01:58:53.766422 730 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:53.745537 659 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:53.747282 771 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:53.770205 30926 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:53.771941 30924 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:53.746592 720 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:53.815224 30965 leader_election.cc:336] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.020s (SENT)
I20250902 01:58:53.784861 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:53.788741 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 108348us
I20250902 01:58:53.780746 757 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:58:53.789530 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 103980us
I20250902 01:58:53.790588 31729 rpcz_store.cc:276] 0902 01:58:52.446728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.684880 (+1238152us) service_pool.cc:225] Handling call
0902 01:58:53.700933 (+ 16053us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.701573 (+ 640us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.701741 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:53.792227 780 consensus_queue.cc:1162] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "01f6731ce5044a6a948a5ae04da80e6f" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 5 index: 6 } }
I20250902 01:58:53.749996 31724 rpcz_store.cc:276] 0902 01:58:51.841634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.120645 (+1279011us) service_pool.cc:225] Handling call
0902 01:58:53.351122 (+230477us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:53.679795 (+328673us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.699647 (+ 19852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:53.793829 714 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:53.795477 30922 rpcz_store.cc:269] 0902 01:58:51.373917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.517987 (+144070us) service_pool.cc:225] Handling call
0902 01:58:53.794958 (+2276971us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54001e0d54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:53.795047 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":561248512}
I20250902 01:58:53.796027 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:53.742342 682 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:53.849562 31726 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:53.796797 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:53.795625 30967 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:53.798502 30998 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:53.897662 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.795442 706 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:53.799758 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.799703 775 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.800719 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:53.801079 778 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250902 01:58:53.801240 693 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:53.801317 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.904050 31070 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:53.801591 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:53.745286 627 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20250902 01:58:53.737942 761 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.758666 532 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:53.846727 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.907155 30924 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:53.907984 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:53.847324 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 135302us
W20250902 01:58:53.796532 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 122758us
I20250902 01:58:53.880398 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.880554 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.887953 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.909286 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.889846 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.886221 687 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.786139 655 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:53.880599 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49670.9us
I20250902 01:58:53.912354 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.813419 771 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:53.831401 720 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:53.915406 771 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:53.917717 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.920156 655 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.877214 31722 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:53.920629 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.921185 780 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:53.846050 757 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:53.921725 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:53.896859 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 101333us
I20250902 01:58:53.902029 706 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:53.809007 730 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:53.909942 30998 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:53.914497 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.914734 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:53.901599 659 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:53.892921 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.934082 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.913239 31695 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:53.910437 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.927496 30892 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:53.923496 31728 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:53.931320 30889 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:53.922839 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.923410 30967 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:53.935474 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:53.914521 773 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250902 01:58:53.910758 660 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:53.888085 682 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:53.988013 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:53.940960 31695 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:53.989136 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:53.989460 31727 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:53.989817 30926 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:53.989763 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49942.2us
W20250902 01:58:53.990077 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 58863.4us
I20250902 01:58:53.990273 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:53.951426 31726 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:53.990916 31729 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:58:53.992372 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 69040.1us
I20250902 01:58:53.997000 30889 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:53.998064 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:53.999934 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:54.001888 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.001981 31695 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:53.995553 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:53.999612 30967 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:53.987604 743 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 1.000s user 0.006s sys 0.004s
I20250902 01:58:53.991501 655 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:54.004633 31695 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:53.987784 669 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.003756 693 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:54.005517 714 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:53.991883 784 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:54.011428 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:53.932507 659 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:53.994536 475 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:53.998665 615 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:53.998816 529 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:53.653223 30999 rpcz_store.cc:276] 0902 01:58:52.147992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.478504 (+330512us) service_pool.cc:225] Handling call
0902 01:58:52.483739 (+ 5235us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:52.484015 (+ 276us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:52.484038 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:52.484423 (+ 385us) log.cc:844] Serialized 26 byte log entry
0902 01:58:52.907492 (+423069us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:53.558980 (+651488us) log.cc:844] Serialized 14 byte log entry
0902 01:58:53.575479 (+ 16499us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.575558 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:53.575628 (+ 70us) raft_consensus.cc:1683] finished
0902 01:58:53.575642 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:53.575800 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16207,"raft.queue_time_us":95124,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":18793,"thread_start_us":1065493,"threads_started":2,"wal-append.queue_time_us":462079}
I20250902 01:58:53.988996 31724 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:54.020437 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.029946 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.020700 532 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.003108 517 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:54.020313 31074 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:58:54.001071 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.025609 30967 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.028224 31727 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:54.033684 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:53.994814 687 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20250902 01:58:54.009274 714 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:54.044697 31073 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:58:54.000700 787 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20250902 01:58:54.000604 660 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20250902 01:58:53.940037 783 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20250902 01:58:54.012583 655 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:54.039337 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 329) took 1137 ms. Trace:
I20250902 01:58:54.038604 525 consensus_queue.cc:1046] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20250902 01:58:54.021481 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.038952 517 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:54.070660 707 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:54.070919 517 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.071182 707 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:54.006770 347 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:54.045360 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:54.015987 693 raft_consensus.cc:695] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:54.044739 714 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:58:54.048683 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:54.025400 659 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:54.061681 706 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.062624 711 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:54.063985 30998 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:54.028532 529 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:54.095687 529 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:54.098315 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1766) took 2725 ms (client timeout 1924 ms). Trace:
W20250902 01:58:54.098650 30923 rpcz_store.cc:269] 0902 01:58:51.372447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.458992 (+ 86545us) service_pool.cc:225] Handling call
0902 01:58:54.098139 (+2639147us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b5400337554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:54.098256 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5046603520}
I20250902 01:58:54.027031 615 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:54.068424 475 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250902 01:58:54.099721 615 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:54.068323 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 309562us
I20250902 01:58:54.068948 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.070231 30994 rpcz_store.cc:276] 0902 01:58:52.901934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.902039 (+ 105us) service_pool.cc:225] Handling call
0902 01:58:54.039160 (+1137121us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b54000b5c54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:54.039274 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38920064}
I20250902 01:58:54.034077 623 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:54.073375 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.011806 743 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:54.074687 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 74850.5us
I20250902 01:58:54.027182 624 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250902 01:58:54.102101 707 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.022022 372 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:58:54.081027 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.072674 655 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.039179 701 consensus_queue.cc:1046] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20250902 01:58:54.051899 31721 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:54.121197 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.074225 347 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:54.093847 762 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:54.093899 693 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:54.125741 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1777) took 2703 ms (client timeout 2004 ms). Trace:
W20250902 01:58:54.126157 30921 rpcz_store.cc:269] 0902 01:58:51.421889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.469633 (+ 47744us) service_pool.cc:225] Handling call
0902 01:58:54.125566 (+2655933us) spinlock_profiling.cc:243] Waited 95.8 ms on lock 0x7b54002a12d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:54.125675 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220252032}
I20250902 01:58:54.102785 623 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:58:54.094791 792 log.cc:927] Time spent T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250902 01:58:54.095278 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1832) took 4561 ms. Trace:
I20250902 01:58:54.129642 30996 rpcz_store.cc:276] 0902 01:58:49.533997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:50.844270 (+1310273us) service_pool.cc:225] Handling call
0902 01:58:50.901794 (+ 57524us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:51.276152 (+374358us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:51.276197 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:51.291554 (+ 15357us) log.cc:844] Serialized 26 byte log entry
0902 01:58:52.799564 (+1508010us) raft_consensus.cc:1643] Marking committed up to 1
0902 01:58:52.799604 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.094926 (+1295322us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54000b6154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:54.094973 (+ 47us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:54.095018 (+ 45us) raft_consensus.cc:1683] finished
0902 01:58:54.095031 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:54.095228 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":248,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":20514,"spinlock_wait_cycles":858711296,"thread_start_us":1424522,"threads_started":1,"wal-append.queue_time_us":1156330,"wal-append.run_cpu_time_us":1389,"wal-append.run_wall_time_us":1006559}
I20250902 01:58:54.075573 700 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:54.100741 30926 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:54.103031 30889 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:54.104118 30994 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:54.076467 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.082901 714 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:54.104334 720 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.110911 372 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:54.141633 372 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.142102 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.111718 707 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:54.107177 662 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.005s
I20250902 01:58:54.114691 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1610) took 1418 ms. Trace:
I20250902 01:58:54.148653 30918 rpcz_store.cc:276] 0902 01:58:52.696020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.688481 (+992461us) service_pool.cc:225] Handling call
0902 01:58:53.695700 (+ 7219us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:53.849231 (+153531us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:53.849273 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:53.887326 (+ 38053us) log.cc:844] Serialized 26 byte log entry
0902 01:58:53.999122 (+111796us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:53.999166 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:53.999242 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:54.114517 (+115275us) raft_consensus.cc:1683] finished
0902 01:58:54.114534 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:54.114644 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17251,"raft.queue_time_us":47321,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":3353,"thread_start_us":243158,"threads_started":2,"wal-append.queue_time_us":144111}
I20250902 01:58:54.115850 31070 raft_consensus.cc:2391] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:54.114208 743 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.117130 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.114305 775 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.118021 761 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.160602 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.161760 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1633) took 1310 ms. Trace:
I20250902 01:58:54.120357 721 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.125262 801 raft_consensus.cc:3053] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:54.165520 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:54.125377 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 545) took 2671 ms (client timeout 1686 ms). Trace:
I20250902 01:58:54.134207 31074 raft_consensus.cc:2391] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:54.124487 347 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.144866 30996 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:54.146410 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:54.166390 31730 rpcz_store.cc:269] 0902 01:58:51.454056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.461154 (+ 7098us) service_pool.cc:225] Handling call
0902 01:58:54.125187 (+2664033us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54006f0754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:54.125316 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2292473728}
I20250902 01:58:54.171624 31727 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:54.151227 30998 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:54.152891 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.174422 31724 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:54.153059 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.154018 31725 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250902 01:58:54.133368 623 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:54.157600 30965 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.160452 706 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.163817 30997 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:54.165627 801 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:54.143913 701 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:54.177994 31038 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:54.185767 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.178321 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.179029 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:54.181164 30924 raft_consensus.cc:2391] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:54.181823 31039 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:54.182044 30965 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.147112 517 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.185575 701 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:54.191908 701 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.184144 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:54.184299 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.184852 720 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.194854 632 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.196715 783 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:54.197736 30889 connection.cc:232] RPC call timeout handler was delayed by -1.2020859180000798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20250902 01:58:54.188957 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.199087 30965 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.190268 30994 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:58:54.185212 771 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.194660 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.194832 788 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:54.194680 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.185205 790 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20250902 01:58:54.201834 783 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.202440 30918 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:54.202193 801 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.177552 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:54.192485 618 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20250902 01:58:54.204360 30996 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:54.205487 30965 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:54.217922 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.190661 660 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:54.209815 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:54.217311 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:54.235935 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.210788 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.212745 31074 raft_consensus.cc:1273] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:54.212945 31729 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:58:54.237243 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.182884 31722 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:54.238344 30994 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:54.177269 689 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250902 01:58:54.192582 700 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:54.234442 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.215307 517 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.196725 721 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.175384 529 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.211455 788 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.241657 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.162106 31071 rpcz_store.cc:276] 0902 01:58:52.851735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.853761 (+ 2026us) service_pool.cc:225] Handling call
0902 01:58:52.879717 (+ 25956us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.922567 (+ 42850us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:52.927050 (+ 4483us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:52.935112 (+ 8062us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:52.940927 (+ 5815us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.161571 (+1220644us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b540036ef54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:54.161610 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:54.161712 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86348032}
I20250902 01:58:54.238327 515 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250902 01:58:54.203804 532 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:54.208086 720 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:54.216593 522 raft_consensus.cc:2762] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250902 01:58:54.252631 803 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20250902 01:58:54.253064 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.268450 30996 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:54.216449 711 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:54.232327 798 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:54.256017 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.139631 762 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:54.263324 31039 leader_election.cc:336] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.618s (SENT)
I20250902 01:58:54.265892 783 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.265568 532 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.270386 771 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.270655 511 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.275435 30998 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:54.244199 31721 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:54.260654 801 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:54.255477 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.284354 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.319195 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:54.271708 529 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.283989 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.323554 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.324172 31074 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:54.259840 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.297225 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.298857 788 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.299822 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.272571 798 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:54.301177 757 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.216619 615 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.306332 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.306510 682 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:54.323623 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:54.327564 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 89717us
I20250902 01:58:54.339823 31730 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:58:54.292212 30994 raft_consensus.cc:1273] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:54.330296 30892 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:54.245385 789 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20250902 01:58:54.336509 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.342434 31000 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:54.342876 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 330) took 1423 ms. Trace:
I20250902 01:58:54.343240 31000 rpcz_store.cc:276] 0902 01:58:52.919540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.919733 (+ 193us) service_pool.cc:225] Handling call
0902 01:58:54.336346 (+1416613us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:54.342796 (+ 6450us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4650752}
I20250902 01:58:54.299324 659 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:54.352792 30920 raft_consensus.cc:1238] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:54.310567 714 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.355157 31000 raft_consensus.cc:1273] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250902 01:58:54.333490 798 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:54.358479 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 153030us
I20250902 01:58:54.362691 30994 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:54.350365 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.368732 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.344197 700 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:54.348248 788 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:54.402621 31001 raft_consensus.cc:2391] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:54.400882 788 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:54.460614 788 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:54.461465 810 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20250902 01:58:54.461323 550 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.358366 622 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250902 01:58:54.468974 532 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.426316 677 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:54.470824 677 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:54.462483 807 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20250902 01:58:54.472893 730 consensus_meta.cc:287] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.535s user 0.007s sys 0.003s
W20250902 01:58:54.599285 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:54.563398 31692 leader_election.cc:336] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.526s (SENT)
I20250902 01:58:54.602324 31692 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:54.458083 655 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.612967 30994 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:58:54.614351 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.616453 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.648803 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 396) took 1663 ms. Trace:
W20250902 01:58:54.649250 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 85896.1us
I20250902 01:58:54.649159 30926 rpcz_store.cc:276] 0902 01:58:52.985386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.992017 (+1006631us) service_pool.cc:225] Handling call
0902 01:58:54.100596 (+108579us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:54.133154 (+ 32558us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.646209 (+513055us) spinlock_profiling.cc:243] Waited 95.3 ms on lock 0x7b5400337550. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:54.648751 (+ 2542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":219160832}
I20250902 01:58:54.564185 624 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20250902 01:58:54.656955 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.657881 30995 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250902 01:58:54.650807 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 288279us
W20250902 01:58:54.650676 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:54.660109 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.660981 30965 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:54.617954 762 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.664840 30994 raft_consensus.cc:1273] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250902 01:58:54.665242 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.667407 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.659128 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:54.620420 707 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250902 01:58:54.752895 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 88019.8us
I20250902 01:58:54.591363 730 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:54.753993 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:54.744210 30922 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:58:54.652786 754 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:54.662675 615 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:54.760488 701 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.568s user 0.009s sys 0.001s
I20250902 01:58:54.765082 757 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:54.805846 714 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.805545 30917 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:54.825421 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.821391 30921 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:54.790508 659 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.856298 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.802663 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:54.880174 31039 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:54.929247 30918 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
W20250902 01:58:54.930420 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 50331.8us
I20250902 01:58:54.931315 31039 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:54.932812 31039 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:54.921319 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.860127 682 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:54.910313 711 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:54.952363 31074 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.010s sys 0.055s
W20250902 01:58:54.959662 31074 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.010s sys 0.055s
W20250902 01:58:54.964597 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 298719us
I20250902 01:58:54.965718 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:54.966451 30995 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.976198 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:54.916476 677 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:54.928352 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:54.921917 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:54.909866 622 consensus_queue.cc:1046] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20250902 01:58:54.882180 701 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:54.959537 711 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:54.950315 662 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:55.082228 31038 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:54.978013 786 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250902 01:58:54.958926 550 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:54.964356 683 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:54.980561 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:55.089959 31073 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:54.929869 660 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20250902 01:58:55.039145 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.083902 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:55.085480 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:55.087801 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 152754us
I20250902 01:58:55.013257 31721 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:55.084818 347 consensus_meta.cc:287] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.905s user 0.006s sys 0.005s
I20250902 01:58:55.081344 711 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:55.118589 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:55.119307 30922 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:55.106021 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.54654e+06us
I20250902 01:58:55.010043 655 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.009377 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:55.107290 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.080822 622 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:55.113757 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 189128us
I20250902 01:58:55.101768 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:55.122476 30964 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.041s (SENT) [suppressed 67 similar messages]
I20250902 01:58:55.089257 677 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:55.148620 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.150277 616 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:55.150848 824 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
W20250902 01:58:55.151530 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 404) took 3028 ms (client timeout 2837 ms). Trace:
W20250902 01:58:55.151863 31728 rpcz_store.cc:269] 0902 01:58:52.122780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.837466 (+714686us) service_pool.cc:225] Handling call
0902 01:58:53.917589 (+1080123us) spinlock_profiling.cc:243] Waited 9.01 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:55.151376 (+1233787us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b54005ee2d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.151470 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32336896}
W20250902 01:58:55.152428 812 log.cc:927] Time spent T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
I20250902 01:58:55.153033 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1628) took 3794 ms. Trace:
I20250902 01:58:55.128032 31071 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:55.153373 31068 rpcz_store.cc:276] 0902 01:58:51.358379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:51.391230 (+ 32851us) service_pool.cc:225] Handling call
0902 01:58:51.391536 (+ 306us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:52.799955 (+1408419us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.152827 (+2352872us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001ebed4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:55.152985 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2519769984}
W20250902 01:58:55.158334 623 consensus_meta.cc:287] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.965s user 0.007s sys 0.000s
I20250902 01:58:55.159334 623 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:55.161178 31073 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:58:55.162477 788 consensus_meta.cc:287] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.701s user 0.004s sys 0.005s
I20250902 01:58:55.164019 30923 raft_consensus.cc:2391] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:55.130146 622 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:55.168056 622 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:55.168670 701 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.169420 622 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:55.170282 623 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.171093 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:55.178258 31001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1755) took 1416 ms (client timeout 1865 ms). Trace:
W20250902 01:58:55.178654 31001 rpcz_store.cc:269] 0902 01:58:53.761443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.920380 (+158937us) service_pool.cc:225] Handling call
0902 01:58:55.178084 (+1257704us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54002adad4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.178201 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1106822400}
I20250902 01:58:55.179731 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:55.181648 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:55.181975 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1617) took 2271 ms. Trace:
I20250902 01:58:55.182265 30920 rpcz_store.cc:276] 0902 01:58:52.910594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.697350 (+786756us) service_pool.cc:225] Handling call
0902 01:58:53.697591 (+ 241us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:54.356863 (+659272us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.181785 (+824922us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54001914d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:55.181918 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1506488704}
I20250902 01:58:55.183393 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.128268 730 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.162925 788 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:55.144966 831 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
W20250902 01:58:55.190621 30999 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.002s sys 0.007s
W20250902 01:58:55.190915 30999 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.002s sys 0.007s
I20250902 01:58:55.191589 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.144790 699 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:55.128343 347 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:55.192135 30920 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 2.
I20250902 01:58:55.197333 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:55.199765 30925 raft_consensus.cc:2391] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:55.246680 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:55.371050 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 184375us
W20250902 01:58:55.371110 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.371215 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.374404 30892 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:55.504902 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 130592us
W20250902 01:58:55.434633 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.189996 701 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.133s
I20250902 01:58:55.436790 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:55.506614 31001 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:55.439323 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:55.455451 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:55.463742 836 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
W20250902 01:58:55.467089 754 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.615s user 0.004s sys 0.004s
I20250902 01:58:55.463768 30992 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:55.467561 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 450) took 2599 ms. Trace:
I20250902 01:58:55.372664 30920 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:58:55.502032 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:58:50.399277 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:58:55.524876 31072 rpcz_store.cc:276] 0902 01:58:52.868192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.869185 (+ 993us) service_pool.cc:225] Handling call
0902 01:58:52.869432 (+ 247us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:52.869708 (+ 276us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:52.869730 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:54.324198 (+1454468us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b2c00113ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:58:54.324515 (+ 317us) log.cc:844] Serialized 26 byte log entry
0902 01:58:54.404896 (+ 80381us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:54.653368 (+248472us) log.cc:844] Serialized 14 byte log entry
0902 01:58:54.752374 (+ 99006us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.752460 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:55.467278 (+714818us) raft_consensus.cc:1683] finished
0902 01:58:55.467365 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.467489 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99561,"prepare.queue_time_us":248324,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":107987,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":8414,"spinlock_wait_cycles":12862848,"thread_start_us":80060,"threads_started":1,"wal-append.queue_time_us":868938}
I20250902 01:58:55.526382 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1787) took 1325 ms. Trace:
I20250902 01:58:55.535946 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1853) took 1388 ms. Trace:
I20250902 01:58:55.536283 30998 rpcz_store.cc:276] 0902 01:58:54.148055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.235702 (+ 87647us) service_pool.cc:225] Handling call
0902 01:58:55.535779 (+1300077us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b54002bd254. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.535884 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89416960}
I20250902 01:58:55.541489 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1861) took 1355 ms. Trace:
I20250902 01:58:55.541849 30994 rpcz_store.cc:276] 0902 01:58:54.186413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.739181 (+552768us) service_pool.cc:225] Handling call
0902 01:58:54.739499 (+ 318us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:54.978852 (+239353us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:54.978881 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:55.082048 (+103167us) log.cc:844] Serialized 26 byte log entry
0902 01:58:55.187193 (+105145us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:55.187250 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.187509 (+ 259us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:55.541273 (+353764us) raft_consensus.cc:1683] finished
0902 01:58:55.541294 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.541416 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12963,"thread_start_us":277943,"threads_started":2,"wal-append.queue_time_us":424643}
I20250902 01:58:55.543154 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:55.527061 30922 rpcz_store.cc:276] 0902 01:58:54.200521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.118272 (+917751us) service_pool.cc:225] Handling call
0902 01:58:55.526317 (+408045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.543526 30994 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
W20250902 01:58:55.544217 31038 leader_election.cc:336] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.618s (SENT)
I20250902 01:58:55.544579 31038 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:55.544476 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.545691 30922 raft_consensus.cc:2408] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250902 01:58:55.551901 31069 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:58:55.520879 31721 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:58:55.583454 30997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1758) took 1811 ms (client timeout 1602 ms). Trace:
W20250902 01:58:55.583850 30997 rpcz_store.cc:269] 0902 01:58:53.771658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.784593 (+ 12935us) service_pool.cc:225] Handling call
0902 01:58:55.583291 (+1798698us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b540004fbd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.583396 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":798197504}
W20250902 01:58:55.702081 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 197662us
W20250902 01:58:55.686538 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 247323us
W20250902 01:58:55.745543 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 201351us
W20250902 01:58:55.646780 30995 consensus_meta.cc:287] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.668s user 0.004s sys 0.005s
I20250902 01:58:55.750793 30995 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:55.741874 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:55.751144 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.751842 30994 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:55.751462 30889 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:55.745313 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 218803us
W20250902 01:58:55.753703 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:55.753947 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.558974 622 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:55.525352 514 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:55.584651 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.761547 30922 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:55.762213 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1790) took 1561 ms. Trace:
I20250902 01:58:55.629181 801 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:55.558898 828 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:55.763619 30922 rpcz_store.cc:276] 0902 01:58:54.201200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.555856 (+1354656us) service_pool.cc:225] Handling call
0902 01:58:55.762145 (+206289us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:55.764297 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1799) took 1479 ms (client timeout 1939 ms). Trace:
I20250902 01:58:55.765969 31001 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:58:55.771260 30923 rpcz_store.cc:269] 0902 01:58:54.284549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.667155 (+382606us) service_pool.cc:225] Handling call
0902 01:58:55.764034 (+1096879us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b540034f8d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.764227 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":973427072}
I20250902 01:58:55.685317 725 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:55.684983 721 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:55.772413 31001 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250902 01:58:55.669147 788 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.702951 347 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.775851 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1771) took 1301 ms. Trace:
I20250902 01:58:55.793835 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:55.793879 31001 rpcz_store.cc:276] 0902 01:58:54.474772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.771822 (+1297050us) service_pool.cc:225] Handling call
0902 01:58:55.772137 (+ 315us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:55.775391 (+ 3254us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.775684 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.801357 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.816318 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1633) took 1157 ms. Trace:
W20250902 01:58:55.818997 842 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250902 01:58:55.819211 701 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:58:55.817634 788 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.002s
W20250902 01:58:55.819766 711 consensus_meta.cc:287] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.687s user 0.009s sys 0.000s
I20250902 01:58:55.820312 711 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:55.823433 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 858218us
W20250902 01:58:55.823987 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 702590us
I20250902 01:58:55.788355 721 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:55.826399 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.830936 30926 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:58:55.831329 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1798) took 1566 ms (client timeout 1924 ms). Trace:
W20250902 01:58:55.831385 30965 connection.cc:232] RPC call timeout handler was delayed by -1.1103969020000477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
W20250902 01:58:55.831784 30926 rpcz_store.cc:269] 0902 01:58:54.264560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.826125 (+1561565us) service_pool.cc:225] Handling call
0902 01:58:55.831275 (+ 5150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.758096 760 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250902 01:58:55.826403 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1762) took 2034 ms. Trace:
I20250902 01:58:55.773741 725 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:55.828639 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 670121us
I20250902 01:58:55.824407 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 557) took 1651 ms. Trace:
I20250902 01:58:55.834508 31729 rpcz_store.cc:276] 0902 01:58:54.173299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.173574 (+ 275us) service_pool.cc:225] Handling call
0902 01:58:54.186830 (+ 13256us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:54.237449 (+ 50619us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.824190 (+1586741us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b54005ea954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:55.824352 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59350400}
I20250902 01:58:55.836488 721 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:55.816744 30922 rpcz_store.cc:276] 0902 01:58:54.658497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.771300 (+1112803us) service_pool.cc:225] Handling call
0902 01:58:55.816242 (+ 44942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.840497 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:55.844796 556 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:55.848407 30917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 399) took 2168 ms (client timeout 1934 ms). Trace:
W20250902 01:58:55.848850 30917 rpcz_store.cc:269] 0902 01:58:53.680026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.683450 (+ 3424us) service_pool.cc:225] Handling call
0902 01:58:55.848249 (+2164799us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003836d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.848356 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2375100288}
I20250902 01:58:55.849285 781 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:55.849861 781 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:55.850314 781 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:55.851560 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:55.850407 30917 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:55.854272 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1785) took 1654 ms. Trace:
I20250902 01:58:55.856324 30964 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:55.858677 30964 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:55.860658 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 106748us
I20250902 01:58:55.861336 30917 rpcz_store.cc:276] 0902 01:58:54.200081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.849757 (+1649676us) service_pool.cc:225] Handling call
0902 01:58:55.850093 (+ 336us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.854025 (+ 3932us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.854219 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.862000 30889 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:55.863514 30889 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:55.864715 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:55.795802 798 consensus_meta.cc:287] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.395s user 0.011s sys 0.000s
I20250902 01:58:55.796950 849 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:55.907622 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 49054.2us
I20250902 01:58:55.915107 30994 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:55.917901 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1866) took 1038 ms. Trace:
I20250902 01:58:55.918381 30994 rpcz_store.cc:276] 0902 01:58:54.879144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.559055 (+679911us) service_pool.cc:225] Handling call
0902 01:58:55.917823 (+358768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.919829 30994 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:55.921464 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1774) took 1271 ms. Trace:
I20250902 01:58:55.922673 30994 rpcz_store.cc:276] 0902 01:58:54.650284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.919383 (+1269099us) service_pool.cc:225] Handling call
0902 01:58:55.919684 (+ 301us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.921187 (+ 1503us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.921402 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.851387 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:55.915480 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1783) took 1715 ms. Trace:
W20250902 01:58:55.924319 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 155473us
I20250902 01:58:55.926578 30926 rpcz_store.cc:276] 0902 01:58:54.199648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.839727 (+1640079us) service_pool.cc:225] Handling call
0902 01:58:55.914999 (+ 75272us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.915166 (+ 167us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.915190 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:55.915233 (+ 43us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:55.915262 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.915311 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.915413 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.928650 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1786) took 1728 ms. Trace:
I20250902 01:58:55.929086 30917 rpcz_store.cc:276] 0902 01:58:54.200313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.861975 (+1661662us) service_pool.cc:225] Handling call
0902 01:58:55.928048 (+ 66073us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.928256 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.928291 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:55.928343 (+ 52us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:55.928386 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.928452 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.928575 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.793318 828 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:55.912886 669 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:55.913584 696 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:55.917518 849 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:55.940758 30964 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:55.941778 30917 raft_consensus.cc:1238] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:55.944984 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1772) took 1469 ms. Trace:
I20250902 01:58:55.944484 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1628) took 1287 ms. Trace:
W20250902 01:58:55.949227 31038 leader_election.cc:336] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.975s (SENT)
I20250902 01:58:55.936688 659 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:55.963210 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:55.996766 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 47574.6us
I20250902 01:58:55.946249 30917 rpcz_store.cc:276] 0902 01:58:54.657231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.930109 (+1272878us) service_pool.cc:225] Handling call
0902 01:58:55.941550 (+ 11441us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.944206 (+ 2656us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.944420 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.925150 798 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:55.944821 696 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:55.945194 669 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:55.999289 696 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.999240 798 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:55.946075 31001 rpcz_store.cc:276] 0902 01:58:54.475080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.798534 (+1323454us) service_pool.cc:225] Handling call
0902 01:58:55.821419 (+ 22885us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.821732 (+ 313us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:55.822588 (+ 856us) log.cc:844] Serialized 14 byte log entry
0902 01:58:55.944363 (+121775us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:55.944421 (+ 58us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:55.944471 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.944541 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.944930 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9879,"prepare.queue_time_us":211,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"thread_start_us":111113,"threads_started":1}
I20250902 01:58:56.000190 721 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.000936 669 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:55.997444 31038 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.004s (SENT)
I20250902 01:58:56.002417 721 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:56.002965 721 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:56.003293 721 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:56.015596 721 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:55.969542 845 log.cc:927] Time spent T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.211s user 0.003s sys 0.000s
I20250902 01:58:55.970988 622 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:55.943353 835 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:56.016352 622 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.016489 835 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.018416 622 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:55.971828 30918 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1795) took 1768 ms (client timeout 1900 ms). Trace:
I20250902 01:58:55.933282 615 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250902 01:58:56.018909 835 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:55.976219 624 raft_consensus.cc:2762] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20250902 01:58:55.977421 760 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:56.020041 760 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.021919 760 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:55.936962 725 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.022929 725 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:55.952832 30919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1632) took 1294 ms (client timeout 1513 ms). Trace:
W20250902 01:58:56.027725 30919 rpcz_store.cc:269] 0902 01:58:54.658268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.851077 (+1192809us) service_pool.cc:225] Handling call
0902 01:58:55.947648 (+ 96571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:55.983299 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.029318 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.029527 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1761) took 2237 ms. Trace:
I20250902 01:58:55.985946 31722 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:56.030561 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.029821 30996 rpcz_store.cc:276] 0902 01:58:53.791803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.269049 (+477246us) service_pool.cc:225] Handling call
0902 01:58:54.269293 (+ 244us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:54.328105 (+ 58812us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:54.328135 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:54.333769 (+ 5634us) log.cc:844] Serialized 26 byte log entry
0902 01:58:54.584496 (+250727us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:54.584548 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.029204 (+1444656us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b54002dafd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:56.029240 (+ 36us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:56.029278 (+ 38us) raft_consensus.cc:1683] finished
0902 01:58:56.029289 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.029477 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":50530,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":45444,"spinlock_wait_cycles":113091840,"thread_start_us":224669,"threads_started":1,"wal-append.queue_time_us":973269}
W20250902 01:58:56.031055 31039 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.900s (SENT)
I20250902 01:58:56.031483 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:55.953364 660 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20250902 01:58:56.032446 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:55.987988 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:56.031517 31039 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:56.018944 30918 rpcz_store.cc:269] 0902 01:58:54.203187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.318918 (+115731us) service_pool.cc:225] Handling call
0902 01:58:55.971652 (+1652734us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b540023f5d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:55.971764 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1072234880}
W20250902 01:58:55.993986 30965 leader_election.cc:336] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.513s (SENT)
I20250902 01:58:56.029831 31730 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
W20250902 01:58:56.034336 30965 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.541s (SENT)
I20250902 01:58:56.034718 30965 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:55.941263 801 raft_consensus.cc:695] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:55.995232 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:56.045115 31727 raft_consensus.cc:2374] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4. Re-sending same reply.
I20250902 01:58:56.003854 30920 raft_consensus.cc:2366] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:58:55.945847 347 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:55.947218 822 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:56.048571 347 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.049460 822 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:56.049854 822 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:56.050678 347 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:56.051714 801 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.004290 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.056176 31725 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:56.056349 822 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.057015 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.058230 822 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:55.945055 849 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:56.055114 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.058964 31730 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:56.059779 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:56.060721 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.061592 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:55.997694 659 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:56.013531 30917 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
W20250902 01:58:55.967226 840 log.cc:927] Time spent T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20250902 01:58:56.015781 514 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:55.988794 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.058162 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.084271 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1854) took 1935 ms. Trace:
I20250902 01:58:56.063032 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1742) took 2318 ms. Trace:
I20250902 01:58:56.084720 31000 rpcz_store.cc:276] 0902 01:58:54.148369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.361153 (+212784us) service_pool.cc:225] Handling call
0902 01:58:54.361467 (+ 314us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:54.369665 (+ 8198us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:54.960719 (+591054us) log.cc:844] Serialized 14 byte log entry
0902 01:58:55.825129 (+864410us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:55.825502 (+ 373us) log.cc:844] Serialized 26 byte log entry
0902 01:58:55.825643 (+ 141us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:55.825699 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.825784 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:56.084054 (+258270us) raft_consensus.cc:1683] finished
0902 01:58:56.084078 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.084211 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":706898,"prepare.queue_time_us":419492,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":766157,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"spinlock_wait_cycles":11394816,"thread_start_us":409363,"threads_started":2,"wal-append.queue_time_us":657222}
I20250902 01:58:56.085132 30993 rpcz_store.cc:276] 0902 01:58:53.744862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.744964 (+ 102us) service_pool.cc:225] Handling call
0902 01:58:53.921916 (+176952us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:53.922241 (+ 325us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:53.922269 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:53.922735 (+ 466us) log.cc:844] Serialized 26 byte log entry
0902 01:58:54.014838 (+ 92103us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:54.014896 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.014970 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:54.112207 (+ 97237us) raft_consensus.cc:1683] finished
0902 01:58:54.112226 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.062826 (+1950600us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000b5750. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:56.062984 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1150,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":7329,"spinlock_wait_cycles":395946368,"thread_start_us":91774,"threads_started":1,"wal-append.queue_time_us":149254,"wal-append.run_cpu_time_us":1653,"wal-append.run_wall_time_us":1036869}
I20250902 01:58:56.088176 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.092743 31000 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:56.093448 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1871) took 1164 ms. Trace:
I20250902 01:58:56.093852 31000 rpcz_store.cc:276] 0902 01:58:54.928586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.087242 (+1158656us) service_pool.cc:225] Handling call
0902 01:58:56.092495 (+ 5253us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.093194 (+ 699us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.093395 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:58:56.064708 31039 leader_election.cc:336] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.924s (SENT)
I20250902 01:58:56.005564 781 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.062052 31730 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:56.074086 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:55.979477 711 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:56.072562 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:56.103848 31727 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:56.074005 659 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:56.083454 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:56.099603 31039 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.939s (SENT)
I20250902 01:58:56.045015 759 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:56.106314 30917 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:55.944773 834 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20250902 01:58:55.945073 847 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
I20250902 01:58:56.089680 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:56.116494 30993 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:56.123965 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1872) took 1187 ms. Trace:
I20250902 01:58:56.124351 30993 rpcz_store.cc:276] 0902 01:58:54.936635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.087528 (+1150893us) service_pool.cc:225] Handling call
0902 01:58:56.116289 (+ 28761us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.123721 (+ 7432us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.123914 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:56.119962 31729 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:58:56.107915 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1776) took 1024 ms. Trace:
I20250902 01:58:56.147212 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:56.148662 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:56.149828 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:56.153471 31039 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.030s (SENT)
W20250902 01:58:56.166518 31693 connection.cc:232] RPC call timeout handler was delayed by -1.6234206490000815s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20250902 01:58:56.167054 31693 connection.cc:232] RPC call timeout handler was delayed by -1.5793206490001239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20250902 01:58:56.167367 31693 connection.cc:232] RPC call timeout handler was delayed by -1.537920649000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
I20250902 01:58:56.111162 514 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:56.167879 31693 connection.cc:232] RPC call timeout handler was delayed by -1.46952064900006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
W20250902 01:58:56.171408 31693 connection.cc:232] RPC call timeout handler was delayed by -1.2967206490001217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20250902 01:58:56.171880 31693 connection.cc:232] RPC call timeout handler was delayed by -1.2877206490001072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20250902 01:58:56.172240 31693 connection.cc:232] RPC call timeout handler was delayed by -1.1563206490001221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20250902 01:58:56.172616 31693 connection.cc:232] RPC call timeout handler was delayed by -1.1536206490001177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
W20250902 01:58:56.172960 31693 connection.cc:232] RPC call timeout handler was delayed by -1.0942206490001354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20250902 01:58:56.167398 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1778) took 1064 ms. Trace:
I20250902 01:58:56.174912 30993 rpcz_store.cc:276] 0902 01:58:55.103250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.125642 (+1022392us) service_pool.cc:225] Handling call
0902 01:58:56.166779 (+ 41137us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.166992 (+ 213us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:56.167028 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.167083 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:56.167132 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.167199 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.167337 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:56.176577 30993 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:56.077618 758 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250902 01:58:56.178859 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1779) took 1071 ms. Trace:
I20250902 01:58:56.179442 30993 rpcz_store.cc:276] 0902 01:58:55.107797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.176026 (+1068229us) service_pool.cc:225] Handling call
0902 01:58:56.176381 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.178557 (+ 2176us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.178799 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:56.181363 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1780) took 1060 ms. Trace:
I20250902 01:58:56.174201 31693 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:56.126372 31000 rpcz_store.cc:276] 0902 01:58:55.083752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.106990 (+1023238us) service_pool.cc:225] Handling call
0902 01:58:56.107306 (+ 316us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.107489 (+ 183us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:56.107524 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.107619 (+ 95us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:56.107664 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.107731 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.107861 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:56.183624 30917 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:56.183780 31693 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:56.184485 30993 rpcz_store.cc:276] 0902 01:58:55.120944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.180456 (+1059512us) service_pool.cc:225] Handling call
0902 01:58:56.180811 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.181005 (+ 194us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:56.181040 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.181093 (+ 53us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:56.181138 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.181202 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.181309 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:56.186002 31693 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:56.187273 31693 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:56.191924 31039 leader_election.cc:336] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.090s (SENT)
I20250902 01:58:56.121562 618 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20250902 01:58:56.111302 800 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250902 01:58:56.125072 725 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20250902 01:58:56.199277 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1863) took 1537 ms. Trace:
I20250902 01:58:56.188117 760 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:56.206821 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1768) took 1866 ms. Trace:
I20250902 01:58:56.203963 31001 rpcz_store.cc:276] 0902 01:58:54.661979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.000574 (+1338595us) service_pool.cc:225] Handling call
0902 01:58:56.000856 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.001015 (+ 159us) raft_consensus.cc:1532] Early marking committed up to index 2
0902 01:58:56.001254 (+ 239us) log.cc:844] Serialized 14 byte log entry
0902 01:58:56.198466 (+197212us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.198673 (+ 207us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:56.198817 (+ 144us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.199092 (+ 275us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.199219 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12916,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":214,"thread_start_us":185653,"threads_started":1}
I20250902 01:58:56.199014 624 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:56.212522 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 176747us
I20250902 01:58:56.213660 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:56.220556 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 289177us
W20250902 01:58:56.225817 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 285170us
I20250902 01:58:56.207288 30997 rpcz_store.cc:276] 0902 01:58:54.340272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.584484 (+1244212us) service_pool.cc:225] Handling call
0902 01:58:55.584706 (+ 222us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.584879 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:58:55.620833 (+ 35954us) log.cc:844] Serialized 14 byte log entry
0902 01:58:56.206383 (+585550us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.206442 (+ 59us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:56.206498 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.206619 (+ 121us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.206761 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11016,"prepare.queue_time_us":44956,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":444,"raft.queue_time_us":42803,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":59,"thread_start_us":574154,"threads_started":2,"wal-append.queue_time_us":466924}
I20250902 01:58:56.229152 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:56.234632 30967 leader_election.cc:336] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.182s (SENT)
W20250902 01:58:56.235769 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 369985us
W20250902 01:58:56.221150 30965 leader_election.cc:336] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.928s (SENT)
I20250902 01:58:56.211058 711 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:56.238451 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:56.240907 31066 raft_consensus.cc:2408] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:58:56.240178 30965 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:56.242187 31068 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20250902 01:58:56.243185 30965 leader_election.cc:336] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.116s (SENT)
I20250902 01:58:56.215400 760 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:56.240283 30889 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:56.218413 857 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:56.238992 705 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:56.249516 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:56.252200 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:56.249743 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:56.263382 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:56.259655 30964 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:56.261839 31068 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:56.267374 31692 leader_election.cc:336] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.560s (SENT)
I20250902 01:58:56.268092 31692 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:56.248405 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:56.254971 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:56.255481 30889 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:56.267333 30993 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:58:56.243121 570 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:56.292025 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:56.294080 31692 leader_election.cc:336] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.610s (SENT)
W20250902 01:58:56.295133 31692 leader_election.cc:336] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.732s (SENT)
I20250902 01:58:56.248368 815 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:56.325354 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:58:56.334585 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 75066.6us
I20250902 01:58:56.239506 514 consensus_queue.cc:1046] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20250902 01:58:56.335455 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 88767.2us
W20250902 01:58:56.334645 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 91494.8us
W20250902 01:58:56.338980 31692 leader_election.cc:336] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.773s (SENT)
I20250902 01:58:56.344336 31692 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:56.257448 857 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:56.252357 760 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:56.382198 347 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:58:56.399924 876 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:56.393184 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:56.407161 30993 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:56.383217 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 53662.5us
I20250902 01:58:56.262761 705 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:56.439086 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 93795.1us
W20250902 01:58:56.448163 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 112525us
W20250902 01:58:56.343719 30965 leader_election.cc:336] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.226s (SENT)
I20250902 01:58:56.449566 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:56.383594 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 44609.2us
I20250902 01:58:56.418046 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1862) took 2213 ms. Trace:
I20250902 01:58:56.481957 30998 rpcz_store.cc:276] 0902 01:58:54.205038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.544674 (+1339636us) service_pool.cc:225] Handling call
0902 01:58:55.544920 (+ 246us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:55.827159 (+282239us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.827202 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:55.829989 (+ 2787us) log.cc:844] Serialized 26 byte log entry
0902 01:58:56.218433 (+388444us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:56.218488 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.218591 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:56.417720 (+199129us) raft_consensus.cc:1683] finished
0902 01:58:56.417749 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.417979 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":281906,"raft.queue_time_us":283003,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":119,"thread_start_us":287349,"threads_started":1,"wal-append.queue_time_us":545118}
I20250902 01:58:56.298422 694 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:56.326495 861 log.cc:927] Time spent T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20250902 01:58:56.522270 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 179398us
I20250902 01:58:56.251367 711 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.262692 822 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.177s
W20250902 01:58:56.584246 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 49851.5us
W20250902 01:58:56.587515 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 243836us
W20250902 01:58:56.592073 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 151469us
I20250902 01:58:56.595017 30964 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:56.439718 857 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:56.324348 570 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:56.521973 705 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:56.314419 781 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:56.404184 878 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:56.525871 760 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:56.641737 31728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.057s sys 0.087s
W20250902 01:58:56.644672 31728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s user 0.057s sys 0.090s
I20250902 01:58:56.601889 615 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:56.527139 834 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20250902 01:58:56.339947 849 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.603153 864 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:58:56.682109 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 494946us
W20250902 01:58:56.682968 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 427586us
I20250902 01:58:56.683228 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:56.683905 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 441848us
I20250902 01:58:56.688089 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1782) took 2488 ms. Trace:
I20250902 01:58:56.688345 30923 rpcz_store.cc:276] 0902 01:58:54.199419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.783697 (+1584278us) service_pool.cc:225] Handling call
0902 01:58:55.784017 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.784234 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:58:56.209441 (+425207us) log.cc:844] Serialized 14 byte log entry
0902 01:58:56.687854 (+478413us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:56.687896 (+ 42us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:58:56.687922 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.687973 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.688053 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":181315,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":28323,"thread_start_us":848984,"threads_started":2}
W20250902 01:58:56.679638 879 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20250902 01:58:56.648592 876 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:56.695981 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:56.683820 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:56.700543 31692 leader_election.cc:336] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.821s (SENT)
I20250902 01:58:56.701043 31692 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:56.618644 694 consensus_queue.cc:1046] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250902 01:58:56.707324 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:56.708259 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:56.655622 870 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:56.718497 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:56.644524 696 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
W20250902 01:58:56.720088 30924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 406) took 2949 ms (client timeout 2080 ms). Trace:
I20250902 01:58:56.693377 615 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:58:56.723151 30924 rpcz_store.cc:269] 0902 01:58:53.770806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.907758 (+136952us) service_pool.cc:225] Handling call
0902 01:58:56.719856 (+2812098us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b540018ef54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:56.720008 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":617940352}
I20250902 01:58:56.726243 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:56.726377 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:56.726819 30997 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:58:56.727671 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:56.728283 30997 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:56.689172 571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.020s sys 0.042s
I20250902 01:58:56.693421 855 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:56.704717 864 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:56.705729 876 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:56.702441 800 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:56.747988 30996 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.006s sys 0.016s
W20250902 01:58:56.749065 30996 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s user 0.006s sys 0.017s
W20250902 01:58:56.769066 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 85345.6us
W20250902 01:58:56.784850 31038 leader_election.cc:336] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.901s (SENT)
I20250902 01:58:56.789495 31038 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:56.790717 31038 leader_election.cc:336] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.912s (SENT)
W20250902 01:58:56.744027 30922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.014s sys 0.040s
W20250902 01:58:56.754606 862 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.456s user 0.000s sys 0.001s
W20250902 01:58:56.730957 571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s user 0.021s sys 0.043s
I20250902 01:58:56.766968 801 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:56.746502 876 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:56.735010 615 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.751464 800 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:56.887583 522 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:56.881894 570 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:57.033838 522 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:57.003590 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 300633us
I20250902 01:58:57.036234 615 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:56.791206 31038 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:57.038957 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 248136us
I20250902 01:58:57.052647 801 raft_consensus.cc:695] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:58:57.053247 31038 leader_election.cc:336] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.924s (SENT)
W20250902 01:58:57.026261 30965 leader_election.cc:336] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.837s (SENT)
I20250902 01:58:57.053797 31072 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:57.054451 801 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:56.855615 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 155138us
W20250902 01:58:56.937717 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 254559us
I20250902 01:58:57.058429 31693 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.032441 515 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:57.059912 694 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:56.810254 556 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:57.060182 515 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:56.868192 659 consensus_meta.cc:287] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.747s user 0.009s sys 0.000s
I20250902 01:58:57.062019 659 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:57.062731 30925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1788) took 2861 ms (client timeout 2090 ms). Trace:
I20250902 01:58:57.063067 515 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.030901 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1769) took 2690 ms. Trace:
I20250902 01:58:57.040252 508 consensus_queue.cc:1046] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20250902 01:58:57.045814 570 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:56.923362 711 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:56.799346 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 558) took 2624 ms. Trace:
W20250902 01:58:56.804445 30922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.014s sys 0.040s
W20250902 01:58:56.874419 31069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1578) took 3078 ms (client timeout 2182 ms). Trace:
I20250902 01:58:57.050486 736 consensus_queue.cc:1046] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:56.791179 30997 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:57.038739 655 consensus_queue.cc:1046] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:56.877736 800 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:58:57.002648 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 230308us
I20250902 01:58:57.051573 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.065711 31000 raft_consensus.cc:2391] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:58:57.093106 30992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1763) took 3299 ms (client timeout 2160 ms). Trace:
I20250902 01:58:57.064987 659 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:57.094815 372 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:58:57.093874 30992 rpcz_store.cc:269] 0902 01:58:53.793205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.992315 (+199110us) service_pool.cc:225] Handling call
0902 01:58:54.142001 (+149686us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff1915ce 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:57.092630 (+2950629us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400126e54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:57.092875 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2976711808}
I20250902 01:58:57.096980 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:57.098075 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1784) took 2897 ms. Trace:
I20250902 01:58:57.098397 623 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:57.098623 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.098912 623 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:57.076565 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:57.089435 721 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.073s user 0.014s sys 0.000s
I20250902 01:58:55.832793 30999 rpcz_store.cc:276] 0902 01:58:53.792929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.304047 (+511118us) service_pool.cc:225] Handling call
0902 01:58:54.304357 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:54.304563 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:58:54.436569 (+132006us) log.cc:844] Serialized 14 byte log entry
0902 01:58:55.825998 (+1389429us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:55.826061 (+ 63us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:58:55.826119 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.826204 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:55.826337 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":658866,"prepare.queue_time_us":314243,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":635587,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"thread_start_us":848936,"threads_started":2,"wal-append.queue_time_us":1118953}
I20250902 01:58:57.101047 508 raft_consensus.cc:3053] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:57.101714 508 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.095342 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.102497 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:57.063107 30925 rpcz_store.cc:269] 0902 01:58:54.200724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.934384 (+733660us) service_pool.cc:225] Handling call
0902 01:58:57.062257 (+2127873us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540018db54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:57.062504 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":504512896}
W20250902 01:58:57.064972 30920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1627) took 2407 ms (client timeout 2175 ms). Trace:
I20250902 01:58:57.084151 31693 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:57.093319 31069 rpcz_store.cc:269] 0902 01:58:53.796409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:53.796521 (+ 112us) service_pool.cc:225] Handling call
0902 01:58:56.874147 (+3077626us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400201054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:56.874301 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3806590848}
I20250902 01:58:57.103394 30999 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:57.053824 31038 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:57.108929 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.099419 623 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:57.108425 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.102504 31723 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.108458 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.086128 624 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:57.204625 624 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:57.205899 705 consensus_meta.cc:287] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.006s
I20250902 01:58:56.851886 878 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:56.898998 806 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 2.576s user 0.005s sys 0.031s
I20250902 01:58:57.208562 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.209920 30999 raft_consensus.cc:2391] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.210098 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 556) took 3046 ms. Trace:
I20250902 01:58:57.084736 622 raft_consensus.cc:3053] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:57.211848 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.212226 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:57.090925 619 consensus_queue.cc:1046] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.122s
I20250902 01:58:57.070147 522 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.105206 535 tablet_copy_client.cc:538] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250902 01:58:57.213999 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 160882us
I20250902 01:58:57.213747 705 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.216478 705 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.107818 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.219892 31722 raft_consensus.cc:2374] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5. Re-sending same reply.
I20250902 01:58:57.221046 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.222352 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.223215 31722 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:57.224015 800 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:57.225472 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.226548 31722 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:56.869449 886 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:58:57.227614 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.227942 31730 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:58:56.792292 881 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20250902 01:58:57.228303 31722 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:57.066270 30919 raft_consensus.cc:1238] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:57.106920 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:57.176921 30920 rpcz_store.cc:269] 0902 01:58:54.656974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.525896 (+868922us) service_pool.cc:225] Handling call
0902 01:58:57.064542 (+1538646us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54002a12d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:57.064794 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":567278976}
I20250902 01:58:57.107172 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.103355 508 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.203065 853 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:57.098629 30922 rpcz_store.cc:276] 0902 01:58:54.199872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.840861 (+1640989us) service_pool.cc:225] Handling call
0902 01:58:55.919774 (+ 78913us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.919930 (+ 156us) raft_consensus.cc:1532] Early marking committed up to index 2
0902 01:58:56.211092 (+291162us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.094189 (+883097us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:57.094274 (+ 85us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:57.094431 (+ 157us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:57.097185 (+ 2754us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:57.097406 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":105536,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":3516,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":215,"thread_start_us":1160816,"threads_started":2,"wal-append.queue_time_us":666457}
I20250902 01:58:57.211969 622 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:57.213166 31070 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
W20250902 01:58:57.183735 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 84880.4us
I20250902 01:58:57.202301 31729 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:58:57.100872 721 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.211040 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.244081 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:57.244076 31038 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.939s (SENT)
I20250902 01:58:57.246256 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:56.792804 875 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250902 01:58:57.099649 30995 rpcz_store.cc:276] 0902 01:58:54.340491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.757116 (+1416625us) service_pool.cc:225] Handling call
0902 01:58:55.757496 (+ 380us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:58:55.829312 (+ 71816us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.829343 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:58:55.829871 (+ 528us) log.cc:844] Serialized 52 byte log entry
0902 01:58:56.074462 (+244591us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:58:56.074533 (+ 71us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.074635 (+ 102us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:57.030213 (+955578us) raft_consensus.cc:1683] finished
0902 01:58:57.030276 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:57.030642 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":159860,"raft.queue_time_us":71599,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":5006,"thread_start_us":237532,"threads_started":1,"wal-append.queue_time_us":383251}
I20250902 01:58:57.210598 31726 rpcz_store.cc:276] 0902 01:58:54.163170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.163288 (+ 118us) service_pool.cc:225] Handling call
0902 01:58:54.163541 (+ 253us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:54.164200 (+ 659us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:54.164235 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:54.164676 (+ 441us) log.cc:844] Serialized 26 byte log entry
0902 01:58:54.234179 (+ 69503us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:58:54.234207 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:54.234255 (+ 48us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:56.837309 (+2603054us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:57.209433 (+372124us) raft_consensus.cc:1683] finished
0902 01:58:57.209484 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:57.209872 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8534272,"thread_start_us":69172,"threads_started":1,"wal-append.queue_time_us":107544}
W20250902 01:58:57.250672 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 70216.9us
I20250902 01:58:57.262634 30892 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:57.248662 624 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:56.958158 849 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.291838 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.264958 30992 raft_consensus.cc:2408] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:58:56.900265 828 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:57.296388 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.298074 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.309142 522 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:57.308946 622 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.309871 522 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.310580 622 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:57.303944 31725 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:58:57.318310 620 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:58:57.320189 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.004s sys 0.016s
W20250902 01:58:57.321182 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.004s sys 0.017s
I20250902 01:58:57.322211 898 raft_consensus.cc:3053] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:57.322793 832 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.279s user 0.020s sys 0.065s
W20250902 01:58:57.323242 832 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.335s user 0.020s sys 0.066s
W20250902 01:58:57.322474 30996 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.008s sys 0.012s
I20250902 01:58:57.324111 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:57.325872 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 69171.8us
I20250902 01:58:57.318437 682 consensus_queue.cc:1046] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20250902 01:58:57.328064 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1803) took 2994 ms. Trace:
I20250902 01:58:57.329766 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.315590 31070 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:56.818068 864 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.339627 30999 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.340626 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:57.341851 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:57.341727 876 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.342840 30997 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.318473 31072 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:57.344502 721 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.335726 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:57.345402 508 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.345979 31730 raft_consensus.cc:2391] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:58:57.349032 760 consensus_meta.cc:287] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.694s user 0.010s sys 0.010s
I20250902 01:58:57.347709 30926 rpcz_store.cc:276] 0902 01:58:54.333145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.927548 (+1594403us) service_pool.cc:225] Handling call
0902 01:58:55.938146 (+ 10598us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:55.940504 (+ 2358us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.940545 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:55.943636 (+ 3091us) log.cc:844] Serialized 26 byte log entry
0902 01:58:56.253438 (+309802us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:56.253479 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:56.253539 (+ 60us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:57.327653 (+1074114us) raft_consensus.cc:1683] finished
0902 01:58:57.327695 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:57.327901 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1967,"raft.queue_time_us":213085,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":19168,"thread_start_us":271801,"threads_started":1,"wal-append.queue_time_us":644640}
I20250902 01:58:57.349465 760 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.342840 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:57.350880 30999 raft_consensus.cc:2374] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4. Re-sending same reply.
I20250902 01:58:57.352290 31727 raft_consensus.cc:2366] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:58:57.352420 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.353705 30999 raft_consensus.cc:2391] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:57.355196 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.301105 800 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:58:57.356402 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 115227us
W20250902 01:58:57.357292 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 94752.1us
I20250902 01:58:57.357550 30995 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.359920 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.360733 30995 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.361464 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.358001 800 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:57.362629 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.362866 800 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.364926 800 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.364964 515 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:57.365566 515 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:57.365958 515 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.366142 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.366758 30995 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:58:57.366879 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1779) took 3168 ms (client timeout 1618 ms). Trace:
W20250902 01:58:57.367223 30921 rpcz_store.cc:269] 0902 01:58:54.198700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.203732 (+ 5032us) service_pool.cc:225] Handling call
0902 01:58:57.366487 (+3162755us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54002ca6d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:57.366699 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756935552}
I20250902 01:58:57.367635 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.368466 30995 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:57.368505 616 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:58:57.368876 876 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:57.372409 30993 consensus_meta.cc:287] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.964s user 0.006s sys 0.010s
I20250902 01:58:57.372887 30993 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:57.373044 856 raft_consensus.cc:3053] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:57.373499 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1786) took 1031 ms. Trace:
I20250902 01:58:57.373540 856 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.373965 30993 rpcz_store.cc:276] 0902 01:58:56.342616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.344014 (+ 1398us) service_pool.cc:225] Handling call
0902 01:58:57.373399 (+1029385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:58:57.374946 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:57.375844 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.376529 31000 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:57.359095 30889 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:57.322701 898 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:57.370746 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:57.379076 31038 leader_election.cc:336] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.978s (SENT)
W20250902 01:58:57.390959 30996 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s user 0.009s sys 0.012s
W20250902 01:58:57.320390 857 consensus_meta.cc:287] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.699s user 0.007s sys 0.008s
I20250902 01:58:57.369182 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:57.374871 856 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.398437 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.378780 30889 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:57.351405 760 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.399847 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.404567 31722 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:58:57.406054 874 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:58:57.411226 522 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.419020 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:57.428433 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 344530us
I20250902 01:58:57.433959 30999 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:58:57.430124 620 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:57.419214 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:57.435732 30999 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:57.435617 31725 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:57.369380 687 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20250902 01:58:57.437337 31725 raft_consensus.cc:1238] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250902 01:58:57.411918 866 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s user 0.033s sys 0.042s
I20250902 01:58:57.438257 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:57.438884 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:57.412073 856 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:57.440101 31000 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:57.441761 535 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
I20250902 01:58:57.442094 31729 raft_consensus.cc:2408] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250902 01:58:57.418651 870 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.005s sys 0.032s
I20250902 01:58:57.460198 30995 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
W20250902 01:58:57.462700 31692 leader_election.cc:336] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 2.080s (SENT)
I20250902 01:58:57.378569 898 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.471611 522 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:57.414047 857 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:57.476326 30890 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.010s sys 0.010s
I20250902 01:58:57.456163 874 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250902 01:58:57.477267 870 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.005s sys 0.033s
W20250902 01:58:57.484118 30890 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.011s sys 0.010s
W20250902 01:58:57.489722 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 111145us
I20250902 01:58:57.491328 31729 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20250902 01:58:57.495369 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 137956us
I20250902 01:58:57.503634 31725 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250902 01:58:57.504982 623 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.507117 623 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:57.521111 31723 raft_consensus.cc:2408] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20250902 01:58:57.509593 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 501390us
I20250902 01:58:57.491111 870 raft_consensus.cc:695] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.525727 30999 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250902 01:58:57.527858 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.528332 857 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.529910 30889 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.531852 874 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:57.532259 874 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:57.532594 874 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:57.531213 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.535120 922 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:57.536790 922 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.539012 922 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:57.539696 31069 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250902 01:58:57.531128 623 consensus_queue.cc:1035] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20250902 01:58:57.542198 31692 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.543179 622 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:57.543771 622 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.543655 870 consensus_queue.cc:237] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.545702 622 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:57.521733 31723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 403) took 5399 ms (client timeout 2056 ms). Trace:
I20250902 01:58:57.548987 858 consensus_queue.cc:1046] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:57.551723 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:57.556443 898 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:57.571475 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:57.561457 909 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.233s user 0.002s sys 0.000s
I20250902 01:58:57.575700 922 consensus_queue.cc:1046] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
W20250902 01:58:57.593699 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 59181.6us
I20250902 01:58:57.594168 857 raft_consensus.cc:3053] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:57.594693 857 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:57.595028 31692 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.596382 922 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:57.597225 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 832) took 1596 ms. Trace:
I20250902 01:58:57.598374 31721 rpcz_store.cc:276] 0902 01:58:55.984568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.090628 (+106060us) service_pool.cc:225] Handling call
0902 01:58:56.090956 (+ 328us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.091197 (+ 241us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:56.292210 (+201013us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.069928 (+777718us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b18003b30e0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cb805 00007fc602128020 00007fc60213a6c6 00007fc60214b237 00007fc60214b1b8
0902 01:58:57.461984 (+392056us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.491344 (+ 29360us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:57.499173 (+ 7829us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:57.516166 (+ 16993us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:57.527361 (+ 11195us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:57.580759 (+ 53398us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":966,"prepare.queue_time_us":785764,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":88,"raft.queue_time_us":170606,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":4416,"spinlock_wait_cycles":16344448,"thread_start_us":771017,"threads_started":4,"wal-append.queue_time_us":561154}
W20250902 01:58:57.752820 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:57.547505 31723 rpcz_store.cc:269] 0902 01:58:52.122463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:52.810593 (+688130us) service_pool.cc:225] Handling call
0902 01:58:55.003719 (+2193126us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff1915ce 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:57.095037 (+2091318us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff19166c 00007fc5ff1915d6 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:57.521623 (+426586us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30789248}
W20250902 01:58:57.783663 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 219441us
I20250902 01:58:57.767927 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:57.784919 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 253809us
I20250902 01:58:57.583967 620 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.582168 757 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:57.780316 927 log.cc:927] Time spent T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
W20250902 01:58:57.796260 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 201408us
I20250902 01:58:57.800123 522 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:57.800794 522 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.803958 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.583382 911 raft_consensus.cc:3053] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:57.575736 515 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.785048 857 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.816614 788 raft_consensus.cc:2953] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:57.816036 857 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:57.813906 31038 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.030s (SENT)
I20250902 01:58:57.820201 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.821163 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:57.821707 31071 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:57.803273 522 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:57.823280 31039 leader_election.cc:336] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.422s (SENT)
I20250902 01:58:57.799865 757 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:57.830834 760 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:58:57.824311 522 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:57.812528 694 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:57.786721 874 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.806327 911 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.835939 31692 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:57.835592 31039 leader_election.cc:336] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.910s (SENT)
I20250902 01:58:57.830292 757 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:57.840288 31038 leader_election.cc:336] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.052s (SENT)
W20250902 01:58:57.844110 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 46388.5us
I20250902 01:58:57.845882 760 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:57.836166 694 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.849076 694 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.847234 760 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:57.850350 760 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.834786 31069 raft_consensus.cc:2953] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:57.857685 31693 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.861922 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.864514 694 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:57.864899 31072 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.873215 886 raft_consensus.cc:3053] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:57.873629 886 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:57.881181 31072 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:57.856233 898 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20250902 01:58:57.882370 30892 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:57.864264 515 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:57.869267 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:57.884141 30889 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:58:57.886832 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 48462.2us
W20250902 01:58:57.887724 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 80395.2us
I20250902 01:58:57.869587 874 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:57.886633 623 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:58:57.893512 31038 leader_election.cc:336] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.090s (SENT)
I20250902 01:58:57.874536 886 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.895905 31038 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:57.897082 31692 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.893173 515 raft_consensus.cc:2762] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250902 01:58:57.894418 623 raft_consensus.cc:695] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.904098 522 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:57.906867 886 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:57.892447 911 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.907464 760 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:57.923224 31692 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:57.925917 898 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:57.927412 911 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:57.926972 623 consensus_queue.cc:237] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:57.944430 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:57.939528 857 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:57.949038 937 log.cc:927] Time spent T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20250902 01:58:57.946861 898 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.961304 870 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250902 01:58:57.957769 935 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250902 01:58:57.960009 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:57.962383 788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:57726 (request call id 12) took 5085 ms. Trace:
I20250902 01:58:57.964740 870 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:57.969835 870 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:57.970336 31071 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:57.969641 788 rpcz_store.cc:276] 0902 01:58:52.872757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.019737 (+1146980us) service_pool.cc:225] Handling call
0902 01:58:54.662783 (+643046us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b54001f8220. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c4c50 00007fc602136af2 00007fc6021357b0 00007fc60708e8d9 00007fc600720bc4 00007fc600720b52 00007fc600720a81 00007fc6007209fc 00007fc60071fcb2 00007fc60052fccc
0902 01:58:54.860874 (+198091us) log.cc:844] Serialized 506 byte log entry
0902 01:58:56.260475 (+1399601us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b54001f8220. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c8256 00007fc6020c1bca 00007fc6020c0e34 00007fc6020a9502 00007fc6020ae911 00007fc6020ae899 00007fc6020ae829 00007fc6020ae7f1 00007fc6020ad9bd 0000000000379254
0902 01:58:56.908146 (+647671us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54001f8220. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c086f 00007fc6020a9502 00007fc6020ae911 00007fc6020ae899 00007fc6020ae829 00007fc6020ae7f1 00007fc6020ad9bd 0000000000379254 0000000000379099 00007fc5ff1ce814
0902 01:58:57.073300 (+165154us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b54001f8220. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c086f 00007fc6020a9502 00007fc6020ae911 00007fc6020ae899 00007fc6020ae829 00007fc6020ae7f1 00007fc6020ad9bd 0000000000379254 0000000000379099 00007fc5ff1ce814
0902 01:58:57.957580 (+884280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24371,"raft.queue_time_us":132138,"raft.run_cpu_time_us":81003,"raft.run_wall_time_us":5250209,"spinlock_wait_cycles":106399616,"thread_start_us":154216,"threads_started":1,"wal-append.queue_time_us":280696,"wal-append.run_cpu_time_us":4168,"wal-append.run_wall_time_us":2098616}
I20250902 01:58:57.970856 870 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.971904 31066 raft_consensus.cc:3053] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:57.972519 31066 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:57.974802 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:57.975126 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:57.975811 30926 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:57.978185 30921 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:57.978127 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:57.974001 870 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:57.974849 31066 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:57.978881 31073 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
W20250902 01:58:57.976212 31071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1677) took 1754 ms (client timeout 1916 ms). Trace:
I20250902 01:58:57.986524 31066 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:58:57.987952 31071 rpcz_store.cc:269] 0902 01:58:56.218172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.219671 (+ 1499us) service_pool.cc:225] Handling call
0902 01:58:57.799551 (+1579880us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:58:57.970944 (+171393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70636288}
I20250902 01:58:57.989805 522 consensus_queue.cc:1046] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:57.093595 31724 rpcz_store.cc:276] 0902 01:58:54.175199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:54.183608 (+ 8409us) service_pool.cc:225] Handling call
0902 01:58:55.154229 (+970621us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:55.154670 (+ 441us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:55.154709 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:55.155176 (+ 467us) log.cc:844] Serialized 26 byte log entry
0902 01:58:55.552963 (+397787us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:55.557885 (+ 4922us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:55.576578 (+ 18693us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:55.990397 (+413819us) raft_consensus.cc:1683] finished
0902 01:58:55.994781 (+ 4384us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:56.799021 (+804240us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54005ee2d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:56.799275 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1023,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":2087752960,"thread_start_us":350882,"threads_started":1,"wal-append.queue_time_us":756528,"wal-append.run_cpu_time_us":2020,"wal-append.run_wall_time_us":1332677}
I20250902 01:58:57.991320 789 raft_consensus.cc:3053] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:57.991730 789 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:57.992632 789 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:57.993515 30918 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:57.999207 30921 raft_consensus.cc:1238] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:58.001824 898 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:58:58.015071 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.015825 31038 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.016721 800 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:58.018245 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.020655 800 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.022248 31038 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.024945 800 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.026940 31038 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:58.033337 800 raft_consensus.cc:2762] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250902 01:58:58.034828 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:58.035050 940 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250902 01:58:58.035409 800 consensus_queue.cc:1046] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:58.036329 618 raft_consensus.cc:3053] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:58:58.036736 618 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:58.037809 618 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:58.038810 31039 leader_election.cc:336] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.163s (SENT)
I20250902 01:58:58.039124 618 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.039197 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:58.036350 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.043989 620 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.044911 800 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250902 01:58:58.059159 939 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20250902 01:58:58.082365 757 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.163192 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1787) took 1481 ms. Trace:
I20250902 01:58:58.163573 31001 rpcz_store.cc:276] 0902 01:58:56.681379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.681507 (+ 128us) service_pool.cc:225] Handling call
0902 01:58:56.714879 (+ 33372us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.715088 (+ 209us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:57.026758 (+311670us) log.cc:844] Serialized 14 byte log entry
0902 01:58:58.162467 (+1135709us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b1800401980. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff14665a 00007fc608617409 00007fc5ff007af0 00007fc5ff1cb894 00007fc5ff1cb805 00007fc602128020 00007fc60213a6c6 00007fc60214b237 00007fc60214b1b8
0902 01:58:58.162630 (+ 163us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:58.162713 (+ 83us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:58.162784 (+ 71us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.162879 (+ 95us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:58.163039 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14332,"prepare.queue_time_us":595838,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"spinlock_wait_cycles":39694592,"thread_start_us":563696,"threads_started":2,"wal-append.queue_time_us":526802}
I20250902 01:58:58.234071 788 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:58.234602 788 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.236065 788 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.252346 757 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.252707 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 414) took 2181 ms. Trace:
I20250902 01:58:58.254639 30919 rpcz_store.cc:276] 0902 01:58:56.071085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.074406 (+ 3321us) service_pool.cc:225] Handling call
0902 01:58:57.065774 (+991368us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:57.247973 (+182199us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.252496 (+1004523us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b540018db50. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:58.252620 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2260658560}
I20250902 01:58:58.264809 31066 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250902 01:58:58.265573 31066 pending_rounds.cc:85] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 2
I20250902 01:58:58.266026 31066 pending_rounds.cc:107] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250902 01:58:58.266364 31066 raft_consensus.cc:2887] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:58:58.284159 757 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250902 01:58:58.284770 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:58:58.320803 864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.979s user 0.017s sys 0.013s
W20250902 01:58:58.321163 864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s user 0.017s sys 0.014s
W20250902 01:58:58.332492 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.333925 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:58.334540 30919 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:58:58.340518 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.341200 30922 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:58.342968 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 972571us
I20250902 01:58:58.343787 30964 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.346154 30921 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:58.348803 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.348886 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.349637 30925 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.349697 30919 raft_consensus.cc:2408] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:58.350333 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.350533 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.350808 30925 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:58.351588 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.351791 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.353390 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.353924 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.354103 30924 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:58.355314 30923 raft_consensus.cc:1215] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20250902 01:58:58.355829 30923 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250902 01:58:58.356456 620 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:58.356982 620 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.361933 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.362481 31727 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:58:58.363411 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.351891 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.352598 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.364902 30920 raft_consensus.cc:2374] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
I20250902 01:58:58.365198 30925 raft_consensus.cc:2408] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:58.352844 616 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.366014 616 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.367923 886 consensus_queue.cc:1046] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:58.369104 31695 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:58.369755 760 raft_consensus.cc:3053] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:58.370074 760 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:58.370499 896 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:58:58.370816 896 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:58.371212 896 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:58.370771 760 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.371968 760 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:58.364447 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.379737 31727 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:58.380707 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.359463 620 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.384290 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 432) took 1156 ms. Trace:
I20250902 01:58:58.385727 31725 rpcz_store.cc:276] 0902 01:58:57.227598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:57.438380 (+210782us) service_pool.cc:225] Handling call
0902 01:58:57.438768 (+ 388us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:57.504095 (+ 65327us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.383934 (+879839us) spinlock_profiling.cc:243] Waited 56.5 ms on lock 0x7b540059b854. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:58:58.384198 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":129932032}
I20250902 01:58:58.386927 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.387742 31725 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:58.388200 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.388881 30923 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.389549 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.390043 30923 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:58:58.390808 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.390900 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.391744 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:58.392441 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.392463 31723 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:58.393770 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.394544 30924 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.395349 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.396031 30924 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:58.397593 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.391455 30923 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.391994 30919 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:58.402812 655 raft_consensus.cc:3053] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:58.403928 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.404034 655 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:58:58.404556 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.404867 655 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.405846 870 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.405970 655 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.406443 870 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.408383 870 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.409243 899 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:58:58.408902 870 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.410218 870 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.425460 30889 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.427145 911 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:58:58.427771 870 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.412133 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.428064 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:58.428434 789 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.428655 31066 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:58.428779 30919 raft_consensus.cc:2408] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:58.390846 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.430071 30920 raft_consensus.cc:2408] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250902 01:58:58.430876 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.431207 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.431770 30892 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:58.432821 870 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:58:58.433202 870 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.433625 870 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:58:58.416971 30872 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 088424002a894fd5bcc11a06af111445 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:58:58.417933 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.438061 700 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:58.438549 700 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:58.438951 31693 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:58.403244 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:58.390857 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.434185 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1775) took 3783 ms. Trace:
I20250902 01:58:58.412856 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.440701 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.439909 30994 rpcz_store.cc:276] 0902 01:58:54.650530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:55.925535 (+1275005us) service_pool.cc:225] Handling call
0902 01:58:55.925868 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:55.927684 (+ 1816us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:58:55.939610 (+ 11926us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.113185 (+1173575us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b180041b4e0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d742da 00007fc60213a9da
0902 01:58:58.433756 (+1320571us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:58.433835 (+ 79us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:58.433891 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.433945 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:58.434036 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1119930,"prepare.queue_time_us":11725,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":5850,"spinlock_wait_cycles":6853888,"thread_start_us":183053,"threads_started":1,"wal-append.queue_time_us":1588125}
I20250902 01:58:58.441314 30923 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.441633 31725 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:58.441715 832 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:58.441951 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.442543 30923 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.442229 832 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.443193 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.420835 694 consensus_queue.cc:1046] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:58:58.435424 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1864) took 3556 ms. Trace:
I20250902 01:58:58.424868 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:58.442363 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:58.554433 31692 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.006s sys 0.006s
I20250902 01:58:58.506105 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.546320 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:58:58.465173 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:58.450235 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 47089.7us
I20250902 01:58:58.491952 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:58.655884 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 298367us
I20250902 01:58:58.660058 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.674413 30924 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
W20250902 01:58:58.606324 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 167439us
I20250902 01:58:58.675204 31693 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.675525 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:58.676537 31066 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.496897 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.511801 620 raft_consensus.cc:2802] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:58:58.593745 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.439435 700 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.657138 624 consensus_queue.cc:1046] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250902 01:58:58.658246 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 227232us
I20250902 01:58:58.658350 832 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.659111 615 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:58.660032 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.658608 30922 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:58.557091 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:58.565073 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.663020 30992 raft_consensus.cc:1238] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:58.662681 896 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.663959 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:58.664253 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.664314 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:58.665077 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 321423us
I20250902 01:58:58.665705 616 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.664009 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.499934 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:58:58.668119 906 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.357s user 0.019s sys 0.011s
W20250902 01:58:58.528527 31728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.005s sys 0.005s
W20250902 01:58:58.701642 31728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s user 0.011s sys 0.009s
W20250902 01:58:58.645705 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 238753us
I20250902 01:58:58.701151 30889 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.672664 30926 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:58.538029 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:58.689337 31692 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.721s user 0.009s sys 0.009s
I20250902 01:58:58.688798 31693 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:58.691596 760 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:58.692016 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.692167 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.692664 347 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:58.692746 31072 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:58.692075 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.694675 30921 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
W20250902 01:58:58.690682 30965 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.569s (SENT)
I20250902 01:58:58.695730 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.678689 620 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:58.670452 30996 rpcz_store.cc:276] 0902 01:58:54.878551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.031120 (+1152569us) service_pool.cc:225] Handling call
0902 01:58:56.031336 (+ 216us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:56.031498 (+ 162us) raft_consensus.cc:1532] Early marking committed up to index 4
0902 01:58:56.078359 (+ 46861us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.428697 (+1350338us) log.cc:844] Serialized 14 byte log entry
0902 01:58:58.435065 (+1006368us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:58:58.435126 (+ 61us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:58:58.435180 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.435245 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:58.435380 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":958693,"prepare.queue_time_us":603131,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":112,"raft.queue_time_us":969565,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":17070,"thread_start_us":1316038,"threads_started":2,"wal-append.queue_time_us":796491,"wal-append.run_cpu_time_us":13668,"wal-append.run_wall_time_us":1781134}
I20250902 01:58:58.511360 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 412) took 2446 ms. Trace:
W20250902 01:58:58.704191 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 740033us
I20250902 01:58:58.690732 800 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:58:58.709667 30964 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.716775 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.667194 870 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.710866 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.706858 760 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:58.686961 31721 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:58.727408 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.712797 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
W20250902 01:58:58.715108 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.715602 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.717936 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.714728 30918 rpcz_store.cc:276] 0902 01:58:56.065476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.065638 (+ 162us) service_pool.cc:225] Handling call
0902 01:58:57.375838 (+1310200us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:57.993280 (+617442us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b54003523a0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020bf550 00007fc602132170 00007fc6021331e1 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c
0902 01:58:57.993878 (+ 598us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.511163 (+517285us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54002ca6d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:58.511300 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3090805376}
I20250902 01:58:58.722755 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.728703 30965 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.718663 31695 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.711798 30995 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 5
I20250902 01:58:58.706745 898 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:58.717957 902 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:58.769526 31038 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:58.731084 30925 raft_consensus.cc:2366] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
W20250902 01:58:58.735954 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.736339 522 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:58.733214 886 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:58.736932 31692 leader_election.cc:336] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.613s (SENT)
I20250902 01:58:58.734596 31723 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:58.739874 347 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.740486 30964 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.770002 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.741264 832 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.743678 867 raft_consensus.cc:3053] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:58.745492 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.746331 760 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.796643 760 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:58.748723 896 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.719295 620 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:58.799834 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.800297 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.801066 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.801700 31723 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.750353 30920 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:58.792186 902 raft_consensus.cc:695] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:58.803778 624 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:58.803609 902 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.804183 624 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.804613 624 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:58.730428 616 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:58.751981 31001 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:58:58.808279 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:58.808638 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.809221 31001 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:58.809389 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.753077 954 consensus_queue.cc:1046] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:58:58.754765 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.815436 31730 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:58.728205 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.724200 800 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:58.762753 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.768661 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:58.723722 30992 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:58.751257 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:58.770488 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.819190 30996 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.706933 615 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.826169 615 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.828487 30997 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:58.786001 622 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:58.829525 622 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:58.829898 622 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:58.736469 31725 raft_consensus.cc:2391] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:58.830206 30921 raft_consensus.cc:1238] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:58:58.831830 31725 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:58.783463 922 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:58.772572 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.782305 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:58.740690 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 53493.5us
I20250902 01:58:58.772590 30923 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
W20250902 01:58:58.835521 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 84387.6us
I20250902 01:58:58.793587 30994 raft_consensus.cc:2391] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:58.785748 886 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.791457 867 raft_consensus.cc:738] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:58.793860 857 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250902 01:58:58.794622 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 54284.6us
I20250902 01:58:58.780062 522 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:58.812575 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.785977 30919 raft_consensus.cc:2391] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:58.820755 31730 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:58.811744 618 raft_consensus.cc:3053] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:58.823473 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 339) took 2147 ms. Trace:
I20250902 01:58:58.824538 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.829923 30997 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:58:58.833212 31725 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:58:58.781586 31692 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.737043 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:58.815199 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:58.838316 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:58.843191 30923 raft_consensus.cc:1238] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:58.843766 30995 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:58:58.834509 800 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:58.846279 949 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:58.849071 30998 rpcz_store.cc:276] 0902 01:58:56.675829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:56.677759 (+ 1930us) service_pool.cc:225] Handling call
0902 01:58:56.678140 (+ 381us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:56.696471 (+ 18331us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:56.696506 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:56.711767 (+ 15261us) log.cc:844] Serialized 26 byte log entry
0902 01:58:57.029348 (+317581us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:57.255857 (+226509us) log.cc:844] Serialized 14 byte log entry
0902 01:58:57.313691 (+ 57834us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:57.331607 (+ 17916us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:58.488566 (+1156959us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:58:58.823126 (+334560us) raft_consensus.cc:1683] finished
0902 01:58:58.823202 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:58.823321 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10830,"prepare.queue_time_us":287986,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":1156812,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":18886,"spinlock_wait_cycles":11457408,"thread_start_us":511328,"threads_started":2,"wal-append.queue_time_us":584633}
I20250902 01:58:58.858409 31724 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:58:58.860030 30922 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:58.857864 31727 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:58.886484 31721 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:58:58.858207 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 129637us
I20250902 01:58:58.876386 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:58:58.889671 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 45381.2us
I20250902 01:58:58.893939 30926 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:58:58.895305 31693 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.901721 30998 raft_consensus.cc:2391] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:58.859649 620 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.902606 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.903652 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.903936 620 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:58.867668 876 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:58.904633 622 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.905407 876 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.906392 31729 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:58.906760 622 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.881395 832 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:58.852440 618 raft_consensus.cc:738] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:58.907948 832 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.908500 618 consensus_queue.cc:260] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:58.860790 867 consensus_queue.cc:260] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.909796 618 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:58.910014 832 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.910450 867 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:58.861248 949 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:58.911960 949 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.861284 30994 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:58:58.915988 31039 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.916345 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:58.916977 30994 raft_consensus.cc:2374] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5. Re-sending same reply.
I20250902 01:58:58.917544 31039 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.917975 30994 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:58.919050 30994 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:58.919706 30964 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.920128 30997 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
W20250902 01:58:58.920565 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 68844.8us
I20250902 01:58:58.921264 30964 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.877091 886 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.923195 789 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:58.923633 876 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:58.923947 31721 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:58.924170 876 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.924811 876 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:58.925140 876 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.925491 876 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:58.847105 864 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:58.928231 30994 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:58.928471 30889 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.851749 30921 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20250902 01:58:58.929844 852 consensus_queue.cc:1046] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:58:58.930197 30994 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250902 01:58:58.931852 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 194981us
I20250902 01:58:58.932492 800 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.946372 852 raft_consensus.cc:3053] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:58:58.946728 852 raft_consensus.cc:738] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:58:58.947520 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 171298us
I20250902 01:58:58.947432 800 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:58.934195 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:58.848424 522 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:58.950279 31038 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:58.904606 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:58.875062 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:58.922374 30964 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:58.942183 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 46994.4us
I20250902 01:58:58.943179 30922 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:58:58.943941 30996 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:58.954689 30917 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:58.963196 30996 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:58.963431 30997 raft_consensus.cc:1273] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:58.875126 870 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:58.969190 31693 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:58.881165 347 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:58.976866 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 48471.3us
I20250902 01:58:58.977663 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:58.982185 30889 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:58.879586 624 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:58.961738 622 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:58.964594 620 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:58.965080 864 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:58.958945 832 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:58.968480 905 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:58:59.016722 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 94432.9us
I20250902 01:58:58.987340 912 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.017953 30996 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:58.986502 870 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:58.993880 616 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:58:59.024489 30964 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.017946 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:59.021483 30992 raft_consensus.cc:2408] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250902 01:58:59.029542 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:59.018428 800 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:58.947448 852 consensus_queue.cc:260] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.023540 616 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:59.032625 347 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:58:59.037178 30994 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:58:59.039541 949 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.074355 31723 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:58:59.013231 832 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.032424 864 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20250902 01:58:59.079624 31067 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:58:59.044188 911 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:59.047847 655 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.032s
I20250902 01:58:59.080433 624 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:58:59.033435 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 187964us
I20250902 01:58:59.035892 800 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:59.023195 905 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:59.100939 950 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250902 01:58:59.026657 30819 catalog_manager.cc:5582] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
W20250902 01:58:59.090847 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 66439.5us
W20250902 01:58:59.109931 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:59.045960 616 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:59.042244 899 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:59.035797 30921 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.111824 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 425) took 1906 ms. Trace:
I20250902 01:58:59.094390 615 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.046764 870 consensus_queue.cc:1046] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.062s
I20250902 01:58:59.103153 852 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.100787 903 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:58:59.085073 757 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20250902 01:58:59.108369 31724 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:58:59.108636 800 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:58:59.129037 31692 leader_election.cc:336] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.757s (SENT)
I20250902 01:58:59.152251 31692 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.153111 30964 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:59.160926 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 176006us
I20250902 01:58:59.124361 31728 rpcz_store.cc:276] 0902 01:58:57.205072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:57.316229 (+111157us) service_pool.cc:225] Handling call
0902 01:58:57.343198 (+ 26969us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:57.343566 (+ 368us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:57.343602 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:58.013914 (+670312us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b2c000a5ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:58:58.014311 (+ 397us) log.cc:844] Serialized 26 byte log entry
0902 01:58:58.702019 (+687708us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:58:58.963141 (+261122us) log.cc:844] Serialized 14 byte log entry
0902 01:58:59.108195 (+145054us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:59.108278 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:59.108320 (+ 42us) raft_consensus.cc:1683] finished
0902 01:58:59.111610 (+ 3290us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:59.111763 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13267,"prepare.queue_time_us":325697,"raft.queue_time_us":62257,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":5036,"spinlock_wait_cycles":70342144,"thread_start_us":1060687,"threads_started":3,"wal-append.queue_time_us":121214}
I20250902 01:58:59.160851 30965 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.146164 694 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:58:59.126103 30855 debug-util.cc:398] Leaking SignalData structure 0x7b08001e13a0 after lost signal to thread 29786
I20250902 01:58:59.184358 31038 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.158491 902 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:59.153893 857 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:58:59.191565 30855 debug-util.cc:398] Leaking SignalData structure 0x7b08001ae380 after lost signal to thread 30858
I20250902 01:58:59.187003 30819 catalog_manager.cc:5582] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
W20250902 01:58:59.193015 30855 debug-util.cc:398] Leaking SignalData structure 0x7b08003b3c40 after lost signal to thread 30950
W20250902 01:58:59.194269 30855 debug-util.cc:398] Leaking SignalData structure 0x7b0800378ba0 after lost signal to thread 31025
I20250902 01:58:59.160616 620 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:58:59.195791 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 72127.7us
I20250902 01:58:59.159866 903 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:58:59.165998 912 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:58:59.199942 30855 debug-util.cc:398] Leaking SignalData structure 0x7b08003507e0 after lost signal to thread 31026
W20250902 01:58:59.206892 30855 debug-util.cc:398] Leaking SignalData structure 0x7b0800329ee0 after lost signal to thread 31099
W20250902 01:58:59.207228 30855 debug-util.cc:398] Leaking SignalData structure 0x7b0800044680 after lost signal to thread 31100
W20250902 01:58:59.208607 30855 debug-util.cc:398] Leaking SignalData structure 0x7b0800164f60 after lost signal to thread 31754
W20250902 01:58:59.209682 30855 debug-util.cc:398] Leaking SignalData structure 0x7b080032fe60 after lost signal to thread 858
W20250902 01:58:59.209782 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 160469us
W20250902 01:58:59.210584 30855 debug-util.cc:398] Leaking SignalData structure 0x7b080017ffe0 after lost signal to thread 919
I20250902 01:58:59.126317 615 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.219372 30964 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:58:59.219709 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 90709.1us
I20250902 01:58:59.165242 700 consensus_queue.cc:1046] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20250902 01:58:59.147495 899 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.198544 624 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.215406 857 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:59.238545 957 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20250902 01:58:59.237788 886 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:58:59.253118 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:59.202693 902 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:59.274410 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 55378us
I20250902 01:58:59.274621 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:59.275449 30917 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:59.244611 949 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:59.248814 954 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:59.276674 30964 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.251439 832 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.276079 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:59.279515 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 118767us
I20250902 01:58:59.251307 700 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.276969 30892 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:58:59.278787 958 log.cc:927] Time spent T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20250902 01:58:59.245725 911 raft_consensus.cc:695] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:59.299702 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:58:59.289340 965 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
W20250902 01:58:59.288209 962 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20250902 01:58:59.294562 876 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:58:59.317286 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 133094us
W20250902 01:58:59.317467 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:59.302495 30998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123"
dest_uuid: "975ecd397f5146409baac87e56a302e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53742
I20250902 01:58:59.318101 30998 raft_consensus.cc:604] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Received request to transfer leadership
I20250902 01:58:59.310566 902 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.254796 522 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.314373 30917 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:58:59.318917 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.293063 905 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:59.281417 903 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.303994 789 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.278344 620 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.328409 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:58:59.337321 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 61301.2us
W20250902 01:58:59.338027 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 61458.2us
I20250902 01:58:59.342731 31728 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:58:59.343247 31728 raft_consensus.cc:604] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.343616 31728 raft_consensus.cc:612] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250902 01:58:59.344647 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 48445.8us
I20250902 01:58:59.343447 31039 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.355965 30965 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.360385 30964 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.298439 984 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20250902 01:58:59.357220 31038 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.336634 700 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.363804 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.365536 30993 raft_consensus.cc:2391] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:59.336892 867 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250902 01:58:59.356396 959 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20250902 01:58:59.327227 870 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
W20250902 01:58:59.362753 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 87245.9us
I20250902 01:58:59.347193 974 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.358733 949 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:58:59.326202 978 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20250902 01:58:59.381062 30921 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:59.382359 31693 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:59.382813 30964 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.371974 832 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:59.383478 700 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.388394 30965 leader_election.cc:400] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:58:59.393646 30965 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.388947 899 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.386693 989 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:59.389331 981 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.386827 949 raft_consensus.cc:695] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:58:59.386687 956 log.cc:927] Time spent T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20250902 01:58:59.381040 974 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.391096 624 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.391181 30964 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.391889 990 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:58:59.370890 982 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:58:59.387748 789 consensus_queue.cc:1046] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20250902 01:58:59.398386 31728 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2925593648194df5b7741d448a047401"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "088424002a894fd5bcc11a06af111445"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:58:59.404158 31728 raft_consensus.cc:604] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Received request to transfer leadership to TS 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.404582 31728 raft_consensus.cc:612] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:58:59.406440 700 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:59.406982 700 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:59.407451 700 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.407830 986 consensus_queue.cc:1046] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:58:59.408980 986 raft_consensus.cc:3053] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:59.409320 986 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.410429 30964 leader_election.cc:400] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:58:59.410701 30964 leader_election.cc:403] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:58:59.410246 986 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.411326 986 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:59.401587 832 raft_consensus.cc:695] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.371246 911 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.419883 989 raft_consensus.cc:3053] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:58:59.429077 983 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20250902 01:58:59.429322 989 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:59.430059 989 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.431074 989 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:58:59.412670 876 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.415486 867 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:59.435729 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1688) took 1953 ms. Trace:
I20250902 01:58:59.436543 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:59.437229 30918 raft_consensus.cc:2374] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5. Re-sending same reply.
I20250902 01:58:59.436180 31070 rpcz_store.cc:276] 0902 01:58:57.482219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:57.488228 (+ 6009us) service_pool.cc:225] Handling call
0902 01:58:57.493371 (+ 5143us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:57.504206 (+ 10835us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:57.504246 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:58.508832 (+1004586us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b2c00113ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:58:58.670477 (+161645us) log.cc:844] Serialized 26 byte log entry
0902 01:58:58.985384 (+314907us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:58.985420 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:58.985482 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:59.431598 (+446116us) raft_consensus.cc:1683] finished
0902 01:58:59.431622 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:59.435637 (+ 4015us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4139,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":3933056,"thread_start_us":287436,"threads_started":1,"wal-append.queue_time_us":423846}
I20250902 01:58:59.439599 522 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.439674 949 consensus_queue.cc:237] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.441797 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.442126 993 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:59.442822 30889 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.401983 899 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20250902 01:58:59.434181 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:58:59.434131 30922 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:59.448096 30926 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:58:59.414505 988 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:59.444881 30921 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250902 01:58:59.449221 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 50276.4us
I20250902 01:58:59.451843 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:59.427356 990 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 5
W20250902 01:58:59.459281 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 47562us
W20250902 01:58:59.462261 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 68689.6us
I20250902 01:58:59.462426 31728 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:58:59.463145 31728 raft_consensus.cc:604] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.463019 30965 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.463663 31728 raft_consensus.cc:612] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:58:59.423102 624 consensus_queue.cc:1046] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250902 01:58:59.468132 991 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.464953 867 raft_consensus.cc:695] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.470510 867 consensus_queue.cc:237] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.450455 984 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:58:59.474498 977 raft_consensus.cc:2802] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:59.467070 899 consensus_queue.cc:1046] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.008s
I20250902 01:58:59.475075 977 raft_consensus.cc:695] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:58:59.475699 972 log.cc:927] Time spent T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20250902 01:58:59.475986 977 consensus_queue.cc:237] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.476786 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 577) took 1078 ms. Trace:
I20250902 01:58:59.477124 31730 rpcz_store.cc:276] 0902 01:58:58.398543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:58.849032 (+450489us) service_pool.cc:225] Handling call
0902 01:58:58.911968 (+ 62936us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:58.948095 (+ 36127us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:58.948123 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:58.948567 (+ 444us) log.cc:844] Serialized 26 byte log entry
0902 01:58:59.145953 (+197386us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:58:59.146007 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:59.146093 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:58:59.476588 (+330495us) raft_consensus.cc:1683] finished
0902 01:58:59.476623 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:58:59.476737 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32913,"raft.queue_time_us":35421,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":196653,"threads_started":1,"wal-append.queue_time_us":348062}
I20250902 01:58:59.478142 996 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.478324 30919 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:59.486248 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.486801 30993 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:59.488294 31071 raft_consensus.cc:1273] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:58:59.461966 876 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.479382 535 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250902 01:58:59.478655 996 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.503551 30993 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:58:59.499152 30965 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:58:59.504062 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:59.502625 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 59871.6us
I20250902 01:58:59.504760 31001 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.506737 30964 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.507764 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:59.508369 31725 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.509766 800 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.510727 31067 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:58:59.511686 969 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:58:59.511931 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.511974 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.512684 535 tablet_bootstrap.cc:492] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:58:59.512810 30996 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:59.513392 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.513630 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.499686 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:59.514998 31722 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:59.511422 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.454880 700 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.483568 988 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:59.520433 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.522428 30999 raft_consensus.cc:2366] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7.
I20250902 01:58:59.494680 618 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:59.495092 624 raft_consensus.cc:3053] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:59.527643 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.478166 616 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:58:59.531656 31692 leader_election.cc:336] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 2.100s (SENT)
I20250902 01:58:59.535734 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.527210 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:58:59.532846 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.513442 535 ts_tablet_manager.cc:1397] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 2.072s user 0.017s sys 0.044s
I20250902 01:58:59.515291 969 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:59.533056 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.541914 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.536249 31726 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:59.538350 30997 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:59.533813 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:58:59.541530 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.499730 1009 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.543349 31728 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:58:59.543382 30999 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:59.484545 832 consensus_queue.cc:237] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.501170 876 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.544529 1002 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:58:59.544937 999 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:59.545879 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.518965 31723 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:58:59.518666 993 raft_consensus.cc:695] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:59.554755 30917 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
W20250902 01:58:59.554889 31692 leader_election.cc:336] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 3.183s (SENT)
I20250902 01:58:59.529039 954 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.539044 1000 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:59.525247 852 consensus_queue.cc:1035] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20250902 01:58:59.555775 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.555845 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.556288 992 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:58:59.557260 978 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.568421 999 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.568957 978 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.570344 999 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.570780 978 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.558655 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 584) took 1128 ms. Trace:
I20250902 01:58:59.533188 624 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.564541 31692 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.560333 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:58:59.559823 30889 consensus_peers.cc:489] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 -> Peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Couldn't send request to peer 01f6731ce5044a6a948a5ae04da80e6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ba5aa5dce214071a172ea365440f0dc. This is attempt 1: this message will repeat every 5th retry.
I20250902 01:58:59.567070 30999 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 2.
I20250902 01:58:59.566459 896 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.572299 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.579701 30994 raft_consensus.cc:3058] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.576215 31722 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:58:59.580412 31000 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250902 01:58:59.518616 1001 consensus_queue.cc:1046] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250902 01:58:59.536659 618 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.583249 30889 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.568217 31723 raft_consensus.cc:604] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.591063 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.573547 1000 raft_consensus.cc:695] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.572968 974 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.593667 31692 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.581924 905 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250902 01:58:59.582652 987 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:58:59.594504 30993 raft_consensus.cc:1273] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:58:59.594923 30889 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.584576 757 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20250902 01:58:59.573706 31724 rpcz_store.cc:276] 0902 01:58:58.429792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:58.882365 (+452573us) service_pool.cc:225] Handling call
0902 01:58:59.099520 (+217155us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:59.155002 (+ 55482us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:59.558451 (+403449us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54005ee2d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:58:59.558597 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":487276800}
I20250902 01:58:59.595916 624 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.605945 992 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:58:59.606230 624 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.606449 992 raft_consensus.cc:695] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:59.607350 992 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.609383 1007 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.609881 1007 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.596899 981 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.589193 852 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.584547 31723 raft_consensus.cc:612] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:58:59.612076 1007 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.613420 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.613538 30889 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:58:59.614053 31072 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:59.614318 992 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:58:59.615212 30892 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:59.615976 852 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:59.616384 852 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:58:59.616541 1007 consensus_queue.cc:1046] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:59.616849 852 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.618705 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.619280 31000 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:58:59.563262 988 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.583025 896 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.599395 760 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:58:59.601557 949 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:58:59.591670 997 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20250902 01:58:59.630489 616 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.615962 992 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:59.624982 800 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:59.632939 618 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.633276 31725 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:59.629470 31038 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.620867 1001 raft_consensus.cc:3053] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:58:59.663612 31692 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.667804 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.668505 31066 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:58:59.666828 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.669631 30892 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:58:59.591832 969 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.663571 700 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.667421 1001 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.685139 800 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20250902 01:58:59.726701 852 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.728670 852 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.678526 1002 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:58:59.730134 1002 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.730646 1004 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:58:59.731338 1004 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:58:59.683449 977 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.732461 1004 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.733141 977 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.735074 977 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:59.683969 30992 raft_consensus.cc:2953] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:58:59.663975 949 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.741820 31038 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:59.712409 31692 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.674178 922 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:58:59.671003 989 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:58:59.626566 993 consensus_queue.cc:237] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.678841 1000 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.718405 876 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.716873 996 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.747509 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:59.756472 903 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.743968 997 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:58:59.756353 1001 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.752058 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:59.787227 922 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.765882 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
W20250902 01:58:59.746783 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51576.6us
I20250902 01:58:59.750322 896 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:58:59.792876 981 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250902 01:58:59.809275 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 43490.4us
I20250902 01:58:59.810220 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:58:59.746702 969 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:59.813320 31038 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:58:59.869478 31692 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:58:59.877633 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 64377.5us
I20250902 01:58:59.879978 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.803459 997 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:58:59.882623 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 213084us
I20250902 01:58:59.804584 989 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:58:59.885133 31072 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
W20250902 01:58:59.875620 1005 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250902 01:58:59.812103 800 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:58:59.888590 31726 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:58:59.885808 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:59.886799 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 232226us
I20250902 01:58:59.887310 31066 raft_consensus.cc:1238] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:58:59.890426 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.869386 977 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:58:59.887523 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.893599 31070 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.816273 981 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:58:59.750391 992 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.880784 896 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:58:59.918999 31726 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:58:59.884744 997 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:58:59.916066 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.887889 1001 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:58:59.887217 987 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:58:59.815629 922 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.929239 30994 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:58:59.930655 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:58:59.938572 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 439508us
I20250902 01:58:59.939105 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.939733 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:58:59.939800 31001 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:58:59.940367 30922 raft_consensus.cc:2391] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:58:59.863263 535 raft_consensus.cc:357] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 NON_PARTICIPANT]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:58:59.894726 977 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:58:59.944845 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:58:59.911516 981 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.919266 988 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.946120 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:58:59.950445 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 137128us
I20250902 01:58:59.946270 30965 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:58:59.911006 800 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:58:59.946666 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:58:59.954211 31692 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:58:59.957872 1012 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:58:59.958431 1012 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.959749 535 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:58:59.960364 1012 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.960769 922 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:58:59.960453 535 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:58:59.961939 986 consensus_meta.cc:287] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.550s user 0.009s sys 0.002s
I20250902 01:58:59.962591 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.924481 989 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.965150 989 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:58:59.966183 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:58:59.970968 1026 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:58:59.972005 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.972745 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.973126 31066 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:58:59.935865 789 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20250902 01:58:59.948673 977 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:58:59.996562 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:58:59.997126 31074 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:58:59.990352 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:00.000377 1019 log.cc:927] Time spent T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250902 01:59:00.001178 1013 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
W20250902 01:58:59.991070 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:58:59.991052 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 44906.9us
I20250902 01:59:00.002918 535 ts_tablet_manager.cc:1428] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.458s user 0.011s sys 0.000s
I20250902 01:58:59.948494 990 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:59:00.005879 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:58:59.995914 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.001641 1013 raft_consensus.cc:695] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:00.004848 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 93105.8us
I20250902 01:58:59.929755 896 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:58:59.977401 949 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:00.012782 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 58764.4us
I20250902 01:59:00.012826 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:00.015022 30965 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:00.011037 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 86096us
I20250902 01:59:00.017166 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.018308 30992 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:00.004927 986 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:00.005216 922 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:59:00.021178 30967 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:00.022672 30889 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.014230 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.026291 31068 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:00.027740 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.028398 31724 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:00.029598 1031 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250902 01:59:00.031944 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 80440.2us
I20250902 01:59:00.032104 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.033391 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1887) took 1370 ms. Trace:
I20250902 01:59:00.033789 30995 rpcz_store.cc:276] 0902 01:58:58.663438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:58.860561 (+197123us) service_pool.cc:225] Handling call
0902 01:58:58.860797 (+ 236us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:58.860999 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:58:59.880339 (+1019340us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b180017db20. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cb805 00007fc605b15327 00007fc60213a957 00007fc60214a06f
0902 01:58:59.881352 (+ 1013us) log.cc:844] Serialized 14 byte log entry
0902 01:59:00.033122 (+151770us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:00.033163 (+ 41us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:00.033199 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:00.033251 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:00.033334 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83484,"prepare.queue_time_us":314785,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":3319,"spinlock_wait_cycles":17023360,"thread_start_us":330141,"threads_started":2,"wal-append.queue_time_us":141982}
I20250902 01:59:00.036558 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.037240 31722 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:00.040405 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.041563 31075 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:00.042347 31695 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.043809 31001 raft_consensus.cc:1238] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:00.044296 30926 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:59:00.044512 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.045092 31727 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:00.045825 30924 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:00.047004 30992 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:00.048180 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.050135 30922 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:00.017522 1020 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:00.053607 30925 raft_consensus.cc:2408] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:00.054687 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.060096 31729 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:00.055599 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.060248 31074 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:00.056180 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.061617 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:00.061674 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.038448 990 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:00.013502 876 consensus_queue.cc:1046] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20250902 01:59:00.063482 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.070539 31692 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.077112 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.079142 30993 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:00.063899 1012 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:00.091426 30993 raft_consensus.cc:1273] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250902 01:59:00.125216 30996 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:00.149035 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.154727 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:00.154901 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 77868us
I20250902 01:59:00.155531 31038 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:00.076377 867 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:00.087990 622 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:00.076548 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.069034 922 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20250902 01:59:00.064903 700 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20250902 01:59:00.058292 1009 consensus_queue.cc:1046] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20250902 01:59:00.088483 1020 raft_consensus.cc:3053] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
W20250902 01:59:00.158151 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 115876us
W20250902 01:59:00.202155 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 131722us
W20250902 01:59:00.201908 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 46460.8us
I20250902 01:59:00.154654 899 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.164328 622 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.081835 905 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
W20250902 01:59:00.159852 1040 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250902 01:59:00.145767 978 consensus_queue.cc:1035] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.110s
I20250902 01:59:00.157429 987 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:00.057039 800 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.217579 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:00.164110 949 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:00.211935 903 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:00.220022 899 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:00.236446 1033 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20250902 01:59:00.237634 899 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:00.152657 760 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20250902 01:59:00.229662 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250902 01:59:00.210148 1020 raft_consensus.cc:738] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:00.216296 999 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:00.240479 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:00.220191 622 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.228447 922 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250902 01:59:00.253794 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 43641.1us
I20250902 01:59:00.220984 789 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:00.231309 31011 tablet_copy_service.cc:342] P 975ecd397f5146409baac87e56a302e9: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-92f16b8a15144fcf8807608bdee71123 received from {username='slave'} at 127.0.0.1:54174
I20250902 01:59:00.263082 31011 tablet_copy_service.cc:434] P 975ecd397f5146409baac87e56a302e9: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-92f16b8a15144fcf8807608bdee71123 on tablet 92f16b8a15144fcf8807608bdee71123 with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:00.264335 899 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:00.264739 899 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:00.231511 949 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:00.267756 31070 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:00.156136 1013 consensus_queue.cc:237] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.247282 896 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.248870 999 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:00.256160 922 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.265096 899 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:00.283579 30994 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
W20250902 01:59:00.281399 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 257319us
W20250902 01:59:00.321720 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 272199us
I20250902 01:59:00.279774 1012 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:00.281318 655 raft_consensus.cc:3053] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:59:00.330250 1035 log.cc:927] Time spent T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20250902 01:59:00.268872 903 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20250902 01:59:00.281258 999 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.340272 31729 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:00.333799 30892 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.270357 789 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.350102 31725 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:59:00.289701 922 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.329363 1012 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:00.333981 655 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:00.334774 615 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.358436 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.362028 31001 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:00.363276 31692 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.335909 1022 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:00.359241 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.370528 31692 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.371677 31727 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:00.374614 31695 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:00.364970 1012 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:00.376725 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:00.377339 31727 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.377569 535 ts_tablet_manager.cc:927] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
I20250902 01:59:00.379094 30993 raft_consensus.cc:2391] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:00.379814 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:00.376868 1039 log.cc:927] Time spent T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20250902 01:59:00.382923 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:00.383427 31725 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.380314 30993 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:00.388387 535 tablet_copy_client.cc:323] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:44457
I20250902 01:59:00.396083 31727 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:00.396986 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.325495 949 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.375346 903 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:00.322166 1020 consensus_queue.cc:260] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.380210 1013 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:00.385843 864 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
W20250902 01:59:00.403714 1028 log.cc:927] Time spent T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.251s user 0.000s sys 0.001s
I20250902 01:59:00.373028 1048 consensus_queue.cc:1035] T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250902 01:59:00.399907 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.397528 1000 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:00.424989 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.425333 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:00.408046 1042 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20250902 01:59:00.401623 1041 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.414183 30920 raft_consensus.cc:1238] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:59:00.414393 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.429153 30924 raft_consensus.cc:2391] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:00.414192 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.430034 1046 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:00.430533 30923 raft_consensus.cc:1273] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:00.430490 1046 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.431751 30923 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:59:00.417052 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.435120 30920 raft_consensus.cc:1273] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:00.401556 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:00.425609 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.437147 30993 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:00.413425 31725 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:00.438043 1046 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.439002 31725 raft_consensus.cc:1215] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
W20250902 01:59:00.439853 1043 log.cc:927] Time spent T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
I20250902 01:59:00.439672 31725 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:59:00.442402 922 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.443051 1000 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:00.444944 1000 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.445299 31727 raft_consensus.cc:1238] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
W20250902 01:59:00.446278 1045 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20250902 01:59:00.445762 922 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:00.430554 1038 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20250902 01:59:00.449237 974 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:00.449579 974 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:00.450263 974 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.409610 1049 raft_consensus.cc:3053] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:59:00.471961 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 138439us
I20250902 01:59:00.374886 1022 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.475711 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.476473 30925 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:00.476833 31001 raft_consensus.cc:1273] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:59:00.481590 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.482195 30926 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
W20250902 01:59:00.470718 1039 log.cc:927] Time spent T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250902 01:59:00.427834 864 raft_consensus.cc:695] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:00.483501 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.488063 30924 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:00.414044 1013 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:00.412282 981 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.493436 31001 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250902 01:59:00.493593 30964 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:00.470947 1049 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:00.405473 655 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.453677 622 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.508752 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:00.509269 30995 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.498917 30964 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:00.494535 974 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.461221 1041 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.514717 31072 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:00.515343 30964 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:00.522578 655 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:00.507963 31723 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:00.505821 1001 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:00.517930 990 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:00.535873 30995 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:00.521893 789 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.524860 1055 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.545315 981 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.550771 31069 raft_consensus.cc:1273] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:00.556351 31074 raft_consensus.cc:1273] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:00.558934 31038 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.540131 615 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:00.573669 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:00.577607 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.583647 31069 raft_consensus.cc:2366] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:59:00.537806 1052 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:00.578577 31068 raft_consensus.cc:1238] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []
I20250902 01:59:00.536109 1060 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:00.557060 789 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:00.597910 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.598697 31075 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:00.575740 987 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:00.589107 31692 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.608868 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.532474 864 consensus_queue.cc:237] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:00.579878 857 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:00.611168 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 104875us
W20250902 01:59:00.614743 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 55905us
I20250902 01:59:00.583462 1001 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:00.585953 1060 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.586490 615 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:00.627132 1081 consensus_queue.cc:1046] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20250902 01:59:00.627863 1013 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:00.628618 1013 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:00.628978 1072 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:00.625214 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53742 (request call id 1013) took 1326 ms. Trace:
I20250902 01:59:00.636727 30998 rpcz_store.cc:276] 0902 01:58:59.294153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.294297 (+ 144us) service_pool.cc:225] Handling call
0902 01:59:00.620399 (+1326102us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x7b54003e9a20. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020c4c50 00007fc60212a612 00007fc6070901ba 00007fc60072a1c4 00007fc60072a152 00007fc60072a081 00007fc600729ffc 00007fc6007292b2 00007fc60052fccc 00007fc60052f106
0902 01:59:00.620637 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12310400}
I20250902 01:59:00.615134 922 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:00.564754 1055 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.600076 1000 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20250902 01:59:00.600952 899 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.623323 912 raft_consensus.cc:3053] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:59:00.616622 622 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.626255 1060 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:00.634333 999 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:00.645262 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:00.659638 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 165453us
I20250902 01:59:00.607973 789 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:00.556787 1049 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.616240 800 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250902 01:59:00.671029 31723 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250902 01:59:00.673079 31075 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:00.596302 1052 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:00.673121 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 49122.4us
I20250902 01:59:00.673025 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.675805 30998 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:00.659613 922 raft_consensus.cc:695] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:00.658890 912 raft_consensus.cc:738] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:00.677402 31692 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.662406 1022 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.677601 1001 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.690492 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:00.692027 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1825) took 1259 ms. Trace:
I20250902 01:59:00.692452 30917 rpcz_store.cc:276] 0902 01:58:59.432882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.433020 (+ 138us) service_pool.cc:225] Handling call
0902 01:58:59.449964 (+ 16944us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:59.564199 (+114235us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:00.691808 (+1127609us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000c26d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:00.691974 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":240504704}
I20250902 01:59:00.694720 31038 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.701021 899 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.701663 899 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250902 01:59:00.704123 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:00.681948 857 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:00.702018 899 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.711697 1077 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:00.711341 899 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:00.713435 1084 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:00.713835 1084 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:00.714143 1084 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.711854 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.717703 1088 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.688997 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.719954 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.727646 30967 leader_election.cc:400] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:59:00.728062 30967 leader_election.cc:403] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20250902 01:59:00.729411 1070 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.109s user 0.000s sys 0.004s
I20250902 01:59:00.729199 1022 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:59:00.730621 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:00.731269 30992 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:00.733176 1008 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 1.107s user 0.004s sys 0.007s
I20250902 01:59:00.735083 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:00.736186 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 444) took 1292 ms. Trace:
I20250902 01:59:00.735785 922 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.736555 30918 rpcz_store.cc:276] 0902 01:58:59.444283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.445057 (+ 774us) service_pool.cc:225] Handling call
0902 01:58:59.451593 (+ 6536us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:58:59.456044 (+ 4451us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:58:59.456068 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:58:59.473239 (+ 17171us) log.cc:844] Serialized 26 byte log entry
0902 01:58:59.538832 (+ 65593us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:58:59.538888 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:58:59.538949 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:00.732558 (+1193609us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:00.735929 (+ 3371us) raft_consensus.cc:1683] finished
0902 01:59:00.735948 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:00.736118 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16976,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"spinlock_wait_cycles":2597632,"thread_start_us":59640,"threads_started":1,"wal-append.queue_time_us":93685}
I20250902 01:59:00.740722 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1715) took 1254 ms. Trace:
I20250902 01:59:00.741108 1052 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:00.741048 31067 rpcz_store.cc:276] 0902 01:58:59.485815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.487255 (+ 1440us) service_pool.cc:225] Handling call
0902 01:58:59.487529 (+ 274us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:59.511043 (+ 23514us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:00.740507 (+1229464us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b5400201554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:00.740654 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52952960}
I20250902 01:59:00.742287 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.742776 31067 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:00.689870 1041 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.743590 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:00.744119 31067 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:00.746249 984 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.746745 984 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.747370 789 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:00.748188 977 consensus_meta.cc:287] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.752s user 0.010s sys 0.000s
I20250902 01:59:00.722520 1055 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.748667 977 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.683881 1000 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20250902 01:59:00.750536 977 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:00.751978 31069 raft_consensus.cc:1273] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:00.757634 31075 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:59:00.759341 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:00.768847 30967 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:00.769366 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:00.716941 857 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.697430 800 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20250902 01:59:00.771790 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:00.787062 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:00.786692 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:00.788723 31066 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:00.782016 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.790105 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.684159 615 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.726279 912 consensus_queue.cc:260] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:00.790432 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.943701 999 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.976490 30998 raft_consensus.cc:2391] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:00.834595 1065 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.756354 832 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.223s
I20250902 01:59:00.829483 1057 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.980187 1057 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.913796 899 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:00.982609 1057 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:00.893693 997 consensus_meta.cc:287] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.954s user 0.007s sys 0.004s
I20250902 01:59:00.827836 30936 tablet_copy_service.cc:140] P 4372b4c4c866431faa1bf3e91fd47a88: Received BeginTabletCopySession request for tablet 4ba5aa5dce214071a172ea365440f0dc from peer 01f6731ce5044a6a948a5ae04da80e6f ({username='slave'} at 127.0.0.1:57742)
I20250902 01:59:00.983814 30936 tablet_copy_service.cc:161] P 4372b4c4c866431faa1bf3e91fd47a88: Beginning new tablet copy session on tablet 4ba5aa5dce214071a172ea365440f0dc from peer 01f6731ce5044a6a948a5ae04da80e6f at {username='slave'} at 127.0.0.1:57742: session id = 01f6731ce5044a6a948a5ae04da80e6f-4ba5aa5dce214071a172ea365440f0dc
I20250902 01:59:00.983741 997 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.985137 30921 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:00.985894 997 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.829291 1026 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:00.988291 1026 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.988801 1026 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:00.989370 31722 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:00.992632 912 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:00.913144 984 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.790532 857 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:00.966972 1084 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.942111 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.996641 31067 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:00.768741 1055 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:00.943987 1012 consensus_meta.cc:287] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.568s user 0.007s sys 0.002s
I20250902 01:59:00.998080 1012 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.786662 1022 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:00.999315 1022 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:00.999449 31074 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:00.999794 1022 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:01.000480 1012 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:01.000813 624 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:01.001276 624 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:01.001147 1055 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.002261 624 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.775451 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:00.803058 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:00.998417 1084 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:01.004678 31001 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:00.828001 985 consensus_queue.cc:1046] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.207s
W20250902 01:59:00.814872 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.005420 985 consensus_queue.cc:1230] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250902 01:59:00.834563 522 raft_consensus.cc:3053] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:01.006217 522 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:01.007133 522 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:00.825752 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:01.009140 30925 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:01.011806 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:00.833752 30997 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:59:01.004091 1055 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:00.833206 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 369) took 1324 ms. Trace:
W20250902 01:59:01.033269 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.032678 30996 rpcz_store.cc:276] 0902 01:58:59.509097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.519490 (+ 10393us) service_pool.cc:225] Handling call
0902 01:59:00.833017 (+1313527us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54000d66d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:00.833127 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1106896640}
W20250902 01:59:01.033661 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:01.034090 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.015864 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:01.018412 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 236495us
I20250902 01:59:01.013037 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250902 01:59:01.035415 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:01.039593 30819 catalog_manager.cc:5582] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:01.046861 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 473786us
I20250902 01:59:01.046965 1091 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250902 01:59:00.838855 1050 log.cc:927] Time spent T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.302s user 0.000s sys 0.000s
I20250902 01:59:01.091770 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:00.803934 903 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:00.834225 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1826) took 1401 ms. Trace:
I20250902 01:59:01.041066 1084 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:01.032866 987 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:01.103801 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 431088us
W20250902 01:59:01.098397 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 57432.5us
I20250902 01:59:00.768496 1088 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:01.032558 905 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:01.114265 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.041815 997 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.092059 624 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:01.138350 31073 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:59:01.102991 832 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:00.842675 1060 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.102090 1052 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.141300 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:01.102210 700 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.140908 30996 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:01.102120 347 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:01.102283 30919 rpcz_store.cc:276] 0902 01:58:59.433229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.443322 (+ 10093us) service_pool.cc:225] Handling call
0902 01:58:59.466178 (+ 22856us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:58:59.478796 (+ 12618us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:00.834006 (+1355210us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b540023ebd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:00.834170 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27543424}
I20250902 01:59:01.134614 899 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:01.246469 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 105244us
I20250902 01:59:01.247638 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:01.240296 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:01.250979 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:01.244292 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:01.259608 31070 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:01.144277 1048 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:01.133417 1091 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.112s
I20250902 01:59:01.137635 905 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.139367 997 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:01.286456 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.238559 949 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.305939 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:01.306631 31001 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:01.280673 30999 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250902 01:59:01.244187 1058 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:01.299645 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:01.232237 911 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:01.249366 967 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20250902 01:59:01.318285 1099 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250902 01:59:01.244377 1072 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:01.146955 832 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.133152 987 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.149968 700 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:01.272226 1000 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.342970 30998 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250902 01:59:01.349543 30998 pending_rounds.cc:85] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 6
I20250902 01:59:01.349797 30998 pending_rounds.cc:107] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250902 01:59:01.350031 30998 raft_consensus.cc:2887] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:01.272372 899 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.150127 1052 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:01.345947 30936 tablet_copy_source_session.cc:215] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Tablet Copy: opened 0 blocks and 1 log segments
I20250902 01:59:01.346503 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:01.347275 1072 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:01.347565 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:01.347287 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:01.360278 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:01.235105 347 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:01.322284 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:01.232386 1001 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:01.272621 992 consensus_queue.cc:1035] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
I20250902 01:59:01.509408 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:01.626313 30891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.018s sys 0.036s
W20250902 01:59:02.129129 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.001s sys 0.001s
W20250902 01:59:02.186327 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.066s user 0.016s sys 0.020s
W20250902 01:59:01.805411 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.713s user 0.028s sys 0.045s
W20250902 01:59:02.435608 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.257s user 0.010s sys 0.001s
W20250902 01:59:02.435825 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.343s user 0.051s sys 0.077s
I20250902 01:59:01.744310 987 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:01.728636 700 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:02.435268 30891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.623s user 0.039s sys 0.090s
I20250902 01:59:02.436939 1113 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:01.599800 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:02.436028 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1839) took 1091 ms. Trace:
W20250902 01:59:02.435637 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.315s user 0.021s sys 0.021s
I20250902 01:59:02.435369 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1925) took 1116 ms. Trace:
W20250902 01:59:02.566671 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 129248us
I20250902 01:59:01.738112 1088 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:02.438066 1113 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.567451 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 457) took 1804 ms. Trace:
I20250902 01:59:01.742014 1060 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:01.350883 1084 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 1.218s
I20250902 01:59:02.569226 1113 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:02.568094 30924 rpcz_store.cc:276] 0902 01:59:00.763106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.763264 (+ 158us) service_pool.cc:225] Handling call
0902 01:59:00.800121 (+ 36857us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:01.049332 (+249211us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:01.049383 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:01.049928 (+ 545us) log.cc:844] Serialized 26 byte log entry
0902 01:59:01.133707 (+ 83779us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:01.133754 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:01.133832 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:02.567214 (+1433382us) raft_consensus.cc:1683] finished
0902 01:59:02.567236 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:02.567388 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":533765,"raft.queue_time_us":36360,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":319358,"threads_started":2,"wal-append.queue_time_us":853541}
W20250902 01:59:02.255373 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.150s user 0.011s sys 0.013s
W20250902 01:59:02.570060 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.465s user 0.014s sys 0.014s
I20250902 01:59:02.570214 1084 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20250902 01:59:02.438313 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.997s user 0.005s sys 0.003s
I20250902 01:59:02.570978 1084 consensus_queue.cc:1230] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer 975ecd397f5146409baac87e56a302e9 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20250902 01:59:02.571169 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.130s user 0.007s sys 0.003s
I20250902 01:59:02.571741 30922 rpcz_store.cc:276] 0902 01:59:01.344424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.344560 (+ 136us) service_pool.cc:225] Handling call
0902 01:59:02.435958 (+1091398us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:02.572273 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:01.398655 922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.043s sys 0.064s
I20250902 01:59:01.392663 30925 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20250902 01:59:01.746766 1069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.018s sys 0.050s
W20250902 01:59:02.574762 1069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.859s user 0.026s sys 0.114s
I20250902 01:59:02.572477 1112 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:01.340384 949 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.737563 997 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:02.575495 1112 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:01.736661 832 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:02.576424 1072 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:02.577082 832 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:02.311729 1034 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s user 0.006s sys 0.007s
I20250902 01:59:01.344689 911 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:01.953966 857 consensus_meta.cc:287] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.958s user 0.003s sys 0.003s
I20250902 01:59:02.531353 1052 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:02.442382 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:02.592949 1099 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:02.341327 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.007s sys 0.002s
I20250902 01:59:02.577855 832 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:01.832535 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.019s sys 0.033s
W20250902 01:59:02.622555 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.850s user 0.021s sys 0.038s
W20250902 01:59:02.622901 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 1.85364e+06us
W20250902 01:59:01.391840 1102 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20250902 01:59:01.906435 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 451) took 1153 ms. Trace:
I20250902 01:59:02.624984 31726 rpcz_store.cc:276] 0902 01:59:00.752840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.752986 (+ 146us) service_pool.cc:225] Handling call
0902 01:59:00.774956 (+ 21970us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:00.775371 (+ 415us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:00.775404 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:00.792557 (+ 17153us) log.cc:844] Serialized 26 byte log entry
0902 01:59:01.096165 (+303608us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:01.096202 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:01.096263 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:01.906194 (+809931us) raft_consensus.cc:1683] finished
0902 01:59:01.906216 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:01.906358 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":538,"raft.queue_time_us":91,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":13785,"thread_start_us":278749,"threads_started":1,"wal-append.queue_time_us":997707}
I20250902 01:59:01.325811 1091 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:01.393024 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:01.558254 31001 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:02.578913 31695 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:02.577647 1112 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:02.570509 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:02.580096 30936 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:57742 (request call id 461) took 1797 ms. Trace:
I20250902 01:59:02.580533 1111 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:01.349341 1058 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:01.953568 624 consensus_meta.cc:287] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.816s user 0.007s sys 0.000s
I20250902 01:59:02.581637 1060 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:02.572825 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:02.441757 620 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.296s user 0.000s sys 0.015s
W20250902 01:59:01.692416 615 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.010s sys 0.024s
I20250902 01:59:02.582010 1113 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:02.582031 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:01.807888 912 consensus_meta.cc:287] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.815s user 0.005s sys 0.005s
I20250902 01:59:01.346513 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:01.803114 1065 consensus_meta.cc:287] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.817s user 0.005s sys 0.004s
W20250902 01:59:02.125136 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.030s user 0.003s sys 0.035s
W20250902 01:59:02.687891 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.593s user 0.006s sys 0.037s
W20250902 01:59:02.688540 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.65323e+06us
I20250902 01:59:02.689325 31693 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:02.672245 620 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.528s user 0.000s sys 0.017s
W20250902 01:59:01.564790 1026 consensus_meta.cc:287] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.576s user 0.009s sys 0.000s
W20250902 01:59:01.563731 31071 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.004s sys 0.025s
W20250902 01:59:02.573087 922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.783s user 0.076s sys 0.134s
W20250902 01:59:01.829548 1022 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.829s user 0.010s sys 0.000s
W20250902 01:59:01.850617 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:01.799439 30926 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.007s sys 0.031s
I20250902 01:59:02.584510 31730 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:01.500795 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 133913us
I20250902 01:59:02.567620 30995 rpcz_store.cc:276] 0902 01:59:01.318576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.344238 (+ 25662us) service_pool.cc:225] Handling call
0902 01:59:02.435310 (+1091072us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:01.405707 899 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:02.482380 31694 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.000s sys 0.016s
W20250902 01:59:02.311966 30992 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.013s sys 0.001s
I20250902 01:59:02.586117 30819 catalog_manager.cc:5582] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:01.286259 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:02.581924 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1838) took 1238 ms. Trace:
I20250902 01:59:02.574492 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1668) took 1231 ms. Trace:
W20250902 01:59:02.125345 864 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.374s user 0.022s sys 0.030s
I20250902 01:59:02.581313 1088 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.577366 949 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:02.577142 1072 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.597057 347 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:02.597252 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 379) took 3081 ms. Trace:
W20250902 01:59:02.601789 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.248s user 0.010s sys 0.010s
I20250902 01:59:02.573282 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:02.447727 31070 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.008s sys 0.001s
W20250902 01:59:02.619963 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s user 0.007s sys 0.006s
W20250902 01:59:02.608081 1034 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.288s user 0.007s sys 0.013s
W20250902 01:59:02.633174 31001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1831) took 1363 ms (client timeout 1706 ms). Trace:
I20250902 01:59:01.393136 1000 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.600414 1001 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:02.632622 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1669) took 1289 ms. Trace:
I20250902 01:59:02.615339 987 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:02.636451 30923 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:02.614619 1103 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:02.639973 30824 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44308 (request call id 328) took 15032 ms (client timeout 15000 ms). Trace:
I20250902 01:59:02.624349 905 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:02.631978 1091 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:02.717363 31071 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.713s user 0.006s sys 0.040s
I20250902 01:59:02.719030 618 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.667320 757 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20250902 01:59:02.665194 1032 consensus_queue.cc:1046] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
W20250902 01:59:02.670066 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:02.634721 1027 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.639938 31723 raft_consensus.cc:2391] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:02.672150 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 870) took 2629 ms. Trace:
I20250902 01:59:02.638979 1112 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.620177 857 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:02.675439 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.428s user 0.001s sys 0.030s
I20250902 01:59:02.663761 1069 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:02.676692 30994 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.357s user 0.020s sys 0.011s
I20250902 01:59:02.639317 992 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:02.672284 615 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.694s user 0.013s sys 0.031s
W20250902 01:59:02.674037 982 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.421s user 0.014s sys 0.016s
I20250902 01:59:02.636963 30936 rpcz_store.cc:276] 0902 01:59:00.803673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.820318 (+ 16645us) service_pool.cc:225] Handling call
0902 01:59:02.580017 (+1759699us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:02.894403 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 456) took 2131 ms. Trace:
I20250902 01:59:02.634887 1099 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:02.728294 30997 rpcz_store.cc:276] 0902 01:58:59.515731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.580683 (+ 64952us) service_pool.cc:225] Handling call
0902 01:59:00.833450 (+1252767us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:01.013679 (+180229us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.597045 (+1583366us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000d66d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:02.597185 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2880895232}
W20250902 01:59:02.742969 1095 log.cc:927] Time spent T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.274s user 0.000s sys 0.002s
I20250902 01:59:02.901149 31728 rpcz_store.cc:276] 0902 01:59:00.042715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.042816 (+ 101us) service_pool.cc:225] Handling call
0902 01:59:01.432531 (+1389715us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:59:01.432716 (+ 185us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:01.432907 (+ 191us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:01.510857 (+ 77950us) log.cc:844] Serialized 14 byte log entry
0902 01:59:02.671817 (+1160960us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:02.671860 (+ 43us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:02.671901 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.671977 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:02.672098 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":78285,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":74246,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"spinlock_wait_cycles":6867712,"thread_start_us":231268,"threads_started":1}
W20250902 01:59:02.719193 31730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 588) took 2793 ms (client timeout 2381 ms). Trace:
I20250902 01:59:02.902545 1020 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:59:02.924602 31730 rpcz_store.cc:269] 0902 01:58:59.925593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.925698 (+ 105us) service_pool.cc:225] Handling call
0902 01:59:01.509222 (+1583524us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:02.719101 (+1209879us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3696512}
W20250902 01:59:02.903553 31723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 878) took 2154 ms (client timeout 1910 ms). Trace:
I20250902 01:59:02.903705 1112 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.717672 1022 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:02.904860 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 530) took 2199 ms. Trace:
W20250902 01:59:02.905174 31694 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.774s user 0.000s sys 0.031s
W20250902 01:59:02.727933 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.374s user 0.012s sys 0.010s
W20250902 01:59:02.934574 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.58115e+06us
I20250902 01:59:02.934367 31067 rpcz_store.cc:276] 0902 01:59:00.705819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.745410 (+ 39591us) service_pool.cc:225] Handling call
0902 01:59:00.745646 (+ 236us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:00.997022 (+251376us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.904618 (+1907596us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400157054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:02.904791 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":576865664}
W20250902 01:59:02.718901 30926 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.744s user 0.007s sys 0.039s
W20250902 01:59:02.669389 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.015s sys 0.053s
I20250902 01:59:02.906157 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1621) took 2345 ms. Trace:
W20250902 01:59:02.728600 31070 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s user 0.008s sys 0.003s
I20250902 01:59:02.906033 30918 rpcz_store.cc:276] 0902 01:59:00.762748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.762886 (+ 138us) service_pool.cc:225] Handling call
0902 01:59:00.771463 (+ 8577us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:00.771759 (+ 296us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:00.771784 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:00.786564 (+ 14780us) log.cc:844] Serialized 26 byte log entry
0902 01:59:01.124332 (+337768us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:01.124384 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:01.124470 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:02.894219 (+1769749us) raft_consensus.cc:1683] finished
0902 01:59:02.894238 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:02.894351 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57261,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":1330304,"thread_start_us":324527,"threads_started":1,"wal-append.queue_time_us":328522}
W20250902 01:59:02.728516 30922 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1843) took 1382 ms (client timeout 1658 ms). Trace:
W20250902 01:59:02.937371 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.363s user 0.016s sys 0.053s
W20250902 01:59:02.909155 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:02.937403 31073 rpcz_store.cc:276] 0902 01:59:00.549098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.549204 (+ 106us) service_pool.cc:225] Handling call
0902 01:59:00.549506 (+ 302us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:01.138814 (+589308us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.756204 (+1617390us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b5400257954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:02.894097 (+137893us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1353657088}
I20250902 01:59:02.912405 30917 rpcz_store.cc:276] 0902 01:59:01.343606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.343724 (+ 118us) service_pool.cc:225] Handling call
0902 01:59:02.632554 (+1288830us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:02.921336 982 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.660s user 0.014s sys 0.016s
W20250902 01:59:02.722711 30919 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1667) took 1379 ms (client timeout 1792 ms). Trace:
I20250902 01:59:02.914379 30925 rpcz_store.cc:276] 0902 01:59:01.343258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.343391 (+ 133us) service_pool.cc:225] Handling call
0902 01:59:01.392252 (+ 48861us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:02.574263 (+1182011us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.574446 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:02.923494 1059 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.819630 989 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:02.690266 31724 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
W20250902 01:59:02.918371 30994 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.599s user 0.022s sys 0.014s
W20250902 01:59:02.918854 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 450) took 3167 ms (client timeout 2033 ms). Trace:
W20250902 01:59:03.029703 30921 rpcz_store.cc:269] 0902 01:58:59.751653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.751781 (+ 128us) service_pool.cc:225] Handling call
0902 01:59:02.738450 (+2986669us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400337554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:02.918761 (+180311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2709143808}
I20250902 01:59:02.746337 1051 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:02.756245 1065 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5.
I20250902 01:59:02.745146 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:02.920157 31001 rpcz_store.cc:269] 0902 01:59:01.269607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.270416 (+ 809us) service_pool.cc:225] Handling call
0902 01:59:02.633094 (+1362678us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:02.939272 615 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:02.933211 993 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:02.750443 832 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:02.722416 1105 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20250902 01:59:02.925150 30997 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:59:02.927110 30819 catalog_manager.cc:5582] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:02.664657 1111 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.938002 857 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.032824 1058 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:02.913419 30992 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.590s user 0.013s sys 0.009s
I20250902 01:59:02.926358 1027 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.032286 1124 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:02.676573 1113 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.671917 1060 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.972251 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:02.728169 1091 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:02.937943 30922 rpcz_store.cc:269] 0902 01:59:01.348474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.572895 (+1224421us) service_pool.cc:225] Handling call
0902 01:59:02.728465 (+155570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:03.065883 1085 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:02.913753 987 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:03.077636 30889 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.624s (SENT) [suppressed 40 similar messages]
I20250902 01:59:02.717410 922 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:02.737022 31729 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:03.067505 905 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.068737 1032 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:02.991250 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1922) took 1694 ms. Trace:
I20250902 01:59:02.933013 1094 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:03.033495 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1834) took 1678 ms. Trace:
I20250902 01:59:03.093919 31001 rpcz_store.cc:276] 0902 01:59:01.354707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.032740 (+1678033us) service_pool.cc:225] Handling call
0902 01:59:03.033009 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:03.033174 (+ 165us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:03.033217 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:03.033275 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:03.033314 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.033366 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.033450 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:02.920871 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.674s user 0.003s sys 0.033s
I20250902 01:59:03.034124 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:03.096050 30925 raft_consensus.cc:2366] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
W20250902 01:59:03.096511 30925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1844) took 1731 ms (client timeout 2008 ms). Trace:
W20250902 01:59:03.096928 30925 rpcz_store.cc:269] 0902 01:59:01.364771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.985294 (+1620523us) service_pool.cc:225] Handling call
0902 01:59:03.096450 (+111156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:03.091833 30819 catalog_manager.cc:5582] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:03.034235 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:02.729688 1072 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.979732 870 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:02.727843 1026 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.731091 1088 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:02.938827 347 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:02.922375 949 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:03.022271 1063 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.377s user 0.020s sys 0.071s
W20250902 01:59:02.927421 31723 rpcz_store.cc:269] 0902 01:59:00.748755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.771458 (+ 22703us) service_pool.cc:225] Handling call
0902 01:59:02.442207 (+1670749us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:02.903454 (+461247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114114304}
I20250902 01:59:02.728163 1103 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.925683 992 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.074806 993 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.926182 1069 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.178439 30924 raft_consensus.cc:2366] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:03.133325 622 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
W20250902 01:59:02.984897 30919 rpcz_store.cc:269] 0902 01:59:01.342921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.346231 (+ 3310us) service_pool.cc:225] Handling call
0902 01:59:02.722655 (+1376424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50560}
W20250902 01:59:02.922097 864 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.171s user 0.027s sys 0.042s
I20250902 01:59:02.911394 30920 rpcz_store.cc:276] 0902 01:59:01.343924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.344132 (+ 208us) service_pool.cc:225] Handling call
0902 01:59:02.581866 (+1237734us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36992}
I20250902 01:59:03.157522 870 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.065991 982 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:03.187804 1126 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.187981 982 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:03.188659 1127 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20250902 01:59:03.190070 982 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.190137 1128 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.027398 1059 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:02.917110 1000 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.173552 31721 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:03.090309 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 877) took 2341 ms. Trace:
I20250902 01:59:02.750751 1110 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:03.074980 1065 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.074569 1079 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:03.098836 30921 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1842) took 1753 ms (client timeout 2241 ms). Trace:
I20250902 01:59:02.728560 618 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.092751 30995 rpcz_store.cc:276] 0902 01:59:01.296922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.924011 (+1627089us) service_pool.cc:225] Handling call
0902 01:59:02.990668 (+ 66657us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:02.990874 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:02.990906 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:02.990970 (+ 64us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:02.991013 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.991076 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:02.991189 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:03.076196 1123 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.034916 989 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.078769 832 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.076277 911 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.156288 30925 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38794
I20250902 01:59:03.100275 1032 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.076402 615 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.157248 31725 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:03.082252 1058 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:03.168182 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:03.168502 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:03.093292 1006 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20250902 01:59:03.173183 949 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:03.175104 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.014s sys 0.034s
W20250902 01:59:03.169051 1118 log.cc:927] Time spent T 6d30dc90c304472ab905d3af8400b8d0 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20250902 01:59:03.185326 1119 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.150691 1034 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:03.168103 1063 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.543s user 0.020s sys 0.071s
I20250902 01:59:02.924273 1001 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:02.727944 30824 rpcz_store.cc:269] 0902 01:58:47.614071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:47.614367 (+ 296us) service_pool.cc:225] Handling call
0902 01:59:02.638546 (+15024179us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:58:48.182649 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:58:48.183121 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:58:48.183158 (+ 37us) write_op.cc:435] Acquired schema lock
0902 01:58:48.183173 (+ 15us) tablet.cc:660] Decoding operations
0902 01:58:48.183812 (+ 639us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:58:48.183877 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0902 01:58:48.183900 (+ 23us) tablet.cc:683] Acquiring locks for 5 operations
0902 01:58:48.184386 (+ 486us) tablet.cc:699] Row locks acquired
0902 01:58:48.184402 (+ 16us) write_op.cc:260] PREPARE: finished
0902 01:58:48.184526 (+ 124us) write_op.cc:270] Start()
0902 01:58:48.184620 (+ 94us) write_op.cc:276] Timestamp: P: 1756778328184486 usec, L: 0
0902 01:58:48.184638 (+ 18us) op_driver.cc:348] REPLICATION: starting
0902 01:58:48.220436 (+ 35798us) log.cc:844] Serialized 2345 byte log entry
0902 01:58:48.501367 (+280931us) op_driver.cc:464] REPLICATION: finished
0902 01:58:48.846524 (+345157us) write_op.cc:301] APPLY: starting
0902 01:58:48.850946 (+ 4422us) tablet.cc:1365] starting BulkCheckPresence
0902 01:58:48.872477 (+ 21531us) tablet.cc:1368] finished BulkCheckPresence
0902 01:58:48.895458 (+ 22981us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:58:49.208508 (+313050us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:58:49.208758 (+ 250us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0902 01:58:49.208824 (+ 66us) write_op.cc:312] APPLY: finished
0902 01:58:49.209108 (+ 284us) log.cc:844] Serialized 47 byte log entry
0902 01:58:49.209285 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:49.209401 (+ 116us) write_op.cc:454] Released schema lock
0902 01:58:49.209449 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0902 01:58:49.209895 (+ 446us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:58:49.209930 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":151039,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":317213,"apply.run_cpu_time_us":3310,"apply.run_wall_time_us":367457,"num_ops":5,"prepare.queue_time_us":127,"prepare.run_cpu_time_us":10945,"prepare.run_wall_time_us":94584,"replication_time_us":287274,"thread_start_us":69898,"threads_started":1}]]}
W20250902 01:59:03.227207 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.020s sys 0.046s
I20250902 01:59:03.077539 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1923) took 1780 ms. Trace:
I20250902 01:59:03.093533 1094 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:03.240329 1104 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.237s user 0.000s sys 0.000s
I20250902 01:59:03.174538 899 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:03.132982 905 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.237118 1091 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:03.168004 347 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.190824 1128 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:01.338934 1067 consensus_queue.cc:1162] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "088424002a894fd5bcc11a06af111445" responder_term: 3 status { last_received { term: 2 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250902 01:59:03.095832 1124 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.151032 1085 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:03.788349 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.098s user 0.029s sys 0.066s
W20250902 01:59:03.789177 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.09999e+06us
W20250902 01:59:03.571293 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.007s
W20250902 01:59:03.789932 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.008s sys 0.014s
I20250902 01:59:03.791769 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1832) took 2521 ms. Trace:
I20250902 01:59:03.792510 535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba5aa5dce214071a172ea365440f0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250902 01:59:03.792107 1123 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.794692 1123 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.795145 1123 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.795630 1137 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.796197 1137 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.796918 989 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.797389 30997 rpcz_store.cc:276] 0902 01:59:01.297206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.924469 (+1627263us) service_pool.cc:225] Handling call
0902 01:59:02.924796 (+ 327us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:03.077312 (+152516us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.077490 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:03.798286 30964 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:03.799118 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1927) took 2431 ms. Trace:
W20250902 01:59:03.603683 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.013s sys 0.009s
W20250902 01:59:03.800289 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.018s sys 0.009s
W20250902 01:59:03.287447 1081 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.659s user 0.052s sys 0.094s
W20250902 01:59:03.800796 1081 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.173s user 0.097s sys 0.155s
W20250902 01:59:03.801543 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.165s user 0.025s sys 0.057s
W20250902 01:59:03.802101 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 1.22328e+06us
I20250902 01:59:03.804220 1139 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:59:03.804976 1124 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:03.805106 1074 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 3.162s user 0.007s sys 0.031s
I20250902 01:59:03.806317 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 387) took 3447 ms. Trace:
W20250902 01:59:03.807590 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.000s sys 0.027s
I20250902 01:59:03.807698 789 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:03.807950 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.000s sys 0.027s
I20250902 01:59:03.806654 30993 rpcz_store.cc:276] 0902 01:59:00.358472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.437717 (+ 79245us) service_pool.cc:225] Handling call
0902 01:59:00.437984 (+ 267us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:00.460953 (+ 22969us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:00.460981 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:00.470872 (+ 9891us) log.cc:844] Serialized 26 byte log entry
0902 01:59:00.534406 (+ 63534us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:00.534440 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:00.534507 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:03.804549 (+3270042us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:03.806046 (+ 1497us) raft_consensus.cc:1683] finished
0902 01:59:03.806068 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.806265 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24574,"raft.queue_time_us":23070,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":11773,"spinlock_wait_cycles":5312256,"thread_start_us":37389,"threads_started":1,"wal-append.queue_time_us":99290}
I20250902 01:59:03.808204 789 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.808912 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:03.809448 30993 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:03.807016 1084 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.810055 789 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.792246 30996 rpcz_store.cc:276] 0902 01:59:01.270284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.270399 (+ 115us) service_pool.cc:225] Handling call
0902 01:59:01.291190 (+ 20791us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:02.669642 (+1378452us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:02.669683 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:02.669730 (+ 47us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:02.669755 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.791552 (+1121797us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54001aead4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:03.791605 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.791716 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4914,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":2045,"spinlock_wait_cycles":3159406976}
I20250902 01:59:03.810410 1084 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.811450 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:03.812325 1084 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.812510 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:03.813411 912 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.241151 1027 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:03.635241 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.007s sys 0.017s
W20250902 01:59:03.814411 31694 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.005s sys 0.021s
W20250902 01:59:03.826081 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.007s sys 0.022s
I20250902 01:59:03.791769 1058 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.789113 30925 raft_consensus.cc:604] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Received request to transfer leadership
I20250902 01:59:03.241036 1099 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.819931 1143 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.820497 1001 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:03.637969 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.015s sys 0.010s
W20250902 01:59:03.631264 30891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.007s sys 0.015s
I20250902 01:59:03.824966 1065 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.820616 30922 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:03.521243 1112 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:03.639608 30992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.008s sys 0.008s
I20250902 01:59:03.822917 1091 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:03.227977 30819 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:03.823873 832 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.478075 1022 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.825299 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 460) took 3051 ms. Trace:
I20250902 01:59:03.799588 30997 rpcz_store.cc:276] 0902 01:59:01.368017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.798277 (+2430260us) service_pool.cc:225] Handling call
0902 01:59:03.798547 (+ 270us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:03.798770 (+ 223us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:03.798792 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:03.798830 (+ 38us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:03.798861 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.798934 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.799048 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:03.813469 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:03.174903 1114 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:03.821748 30921 rpcz_store.cc:269] 0902 01:59:01.345570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.030983 (+1685413us) service_pool.cc:225] Handling call
0902 01:59:03.098778 (+ 67795us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:03.803155 1119 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.188048 870 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.238811 1126 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:03.826282 31694 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.005s sys 0.021s
I20250902 01:59:03.960091 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1914) took 3619 ms. Trace:
I20250902 01:59:03.824355 1110 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.897931 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.203764 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1846) took 1170 ms. Trace:
I20250902 01:59:04.204221 30922 rpcz_store.cc:276] 0902 01:59:03.033651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.093855 (+ 60204us) service_pool.cc:225] Handling call
0902 01:59:04.203682 (+1109827us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:04.120613 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.011s sys 0.010s
W20250902 01:59:04.205307 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.015s sys 0.010s
I20250902 01:59:03.481652 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1626) took 2929 ms. Trace:
W20250902 01:59:04.126549 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 335177us
I20250902 01:59:03.183912 622 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:04.173168 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.233s user 0.021s sys 0.016s
I20250902 01:59:04.205794 31074 rpcz_store.cc:276] 0902 01:59:00.552361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.575698 (+ 23337us) service_pool.cc:225] Handling call
0902 01:59:00.577286 (+ 1588us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:00.999833 (+422547us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.481359 (+2481526us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54000406d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:03.481584 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":970562432}
I20250902 01:59:04.210675 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.211395 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:04.211994 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:04.212469 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:04.213171 1001 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.213554 30921 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:04.214931 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.216220 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:04.216385 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:04.172644 30992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.089s user 0.016s sys 0.020s
I20250902 01:59:03.931756 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:03.755188 1125 log.cc:927] Time spent T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.525s user 0.001s sys 0.000s
I20250902 01:59:04.217617 31069 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:04.218043 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1920) took 2921 ms. Trace:
I20250902 01:59:04.218315 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:04.217355 30997 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
W20250902 01:59:03.726413 30889 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.007s sys 0.005s
W20250902 01:59:04.221335 30889 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.063s user 0.010s sys 0.009s
I20250902 01:59:04.218282 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1628) took 2949 ms. Trace:
I20250902 01:59:03.958547 31067 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:04.213685 1001 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:04.173520 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:04.221592 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.14401e+06us
I20250902 01:59:04.189853 30819 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:04.125749 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 330729us
I20250902 01:59:04.172340 30996 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
W20250902 01:59:03.894528 30994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.000s sys 0.013s
I20250902 01:59:04.221935 31070 rpcz_store.cc:276] 0902 01:59:01.268849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.290358 (+ 21509us) service_pool.cc:225] Handling call
0902 01:59:01.303100 (+ 12742us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:01.327231 (+ 24131us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:01.327274 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:01.388449 (+ 61175us) log.cc:844] Serialized 26 byte log entry
0902 01:59:02.937765 (+1549316us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:02.937812 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.937866 (+ 54us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:04.218112 (+1280246us) raft_consensus.cc:1683] finished
0902 01:59:04.218130 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:04.218235 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10110,"raft.queue_time_us":1167144,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":115,"thread_start_us":1317638,"threads_started":1,"wal-append.queue_time_us":1392190}
W20250902 01:59:03.614327 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.003s sys 0.009s
W20250902 01:59:04.220906 30891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.014s sys 0.022s
W20250902 01:59:03.596065 1098 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.445s user 0.000s sys 0.001s
W20250902 01:59:03.917840 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 119624us
I20250902 01:59:04.138731 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:04.172855 31729 rpcz_store.cc:276] 0902 01:59:00.748485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.752504 (+ 4019us) service_pool.cc:225] Handling call
0902 01:59:02.736569 (+1984065us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:59:02.736802 (+ 233us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:03.090054 (+353252us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:03.090254 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115433344}
I20250902 01:59:04.249028 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38794 (request call id 1116) took 1578 ms. Trace:
I20250902 01:59:04.172873 30926 rpcz_store.cc:276] 0902 01:59:00.774003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.774161 (+ 158us) service_pool.cc:225] Handling call
0902 01:59:00.973250 (+199089us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:00.973937 (+ 687us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:00.973975 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:00.974492 (+ 517us) log.cc:844] Serialized 26 byte log entry
0902 01:59:02.937329 (+1962837us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:02.937367 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.937435 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:03.825026 (+887591us) raft_consensus.cc:1683] finished
0902 01:59:03.825057 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.825229 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"thread_start_us":1962416,"threads_started":1,"wal-append.queue_time_us":1923552}
I20250902 01:59:04.331339 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.172142 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:04.003506 1051 consensus_meta.cc:287] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.968s user 0.007s sys 0.000s
W20250902 01:59:04.126763 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 324002us
I20250902 01:59:04.050551 1004 raft_consensus.cc:3053] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:04.204107 31000 rpcz_store.cc:276] 0902 01:59:00.340664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.344300 (+ 3636us) service_pool.cc:225] Handling call
0902 01:59:00.366689 (+ 22389us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:00.499948 (+133259us) raft_consensus.cc:1532] Early marking committed up to index 9
0902 01:59:00.519683 (+ 19735us) log.cc:844] Serialized 14 byte log entry
0902 01:59:00.519938 (+ 255us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:00.519994 (+ 56us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:00.520044 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.992650 (+2472606us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b18000f1c60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff19c786 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89
0902 01:59:03.959880 (+967230us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000b5c54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:03.959925 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:03.960025 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":76158,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":75,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":3091,"spinlock_wait_cycles":321237504,"thread_start_us":17469,"threads_started":1}
W20250902 01:59:04.205736 31039 connection.cc:232] RPC call timeout handler was delayed by -1.0781586530000595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20250902 01:59:04.344832 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.345429 30997 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:03.757221 1060 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.729269 1109 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.758021 993 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.159861 1063 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:04.203405 31072 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:03.756610 992 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.276811 30918 raft_consensus.cc:2391] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:04.363216 31728 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:04.218063 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:04.219789 30992 rpcz_store.cc:276] 0902 01:59:01.296342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.319417 (+ 23075us) service_pool.cc:225] Handling call
0902 01:59:01.319689 (+ 272us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:01.319868 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:03.083367 (+1763499us) log.cc:844] Serialized 14 byte log entry
0902 01:59:04.217741 (+1134374us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:04.217801 (+ 60us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:04.217853 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:04.217913 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:04.217998 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1094729,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":109,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":2889228,"threads_started":2}
W20250902 01:59:04.335619 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:04.334103 30925 rpcz_store.cc:276] 0902 01:59:02.627609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.133154 (+505545us) service_pool.cc:225] Handling call
0902 01:59:04.205780 (+1072626us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:04.341027 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.246s user 0.005s sys 0.020s
W20250902 01:59:04.001068 1020 consensus_meta.cc:287] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.068s user 0.002s sys 0.008s
I20250902 01:59:04.333911 31066 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:04.341813 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:03.955247 899 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.175951 996 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:03.826226 912 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:03.757027 1088 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:03.807015 1124 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:04.207127 981 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:03.999532 615 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.360906 1091 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.895398 1027 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.360311 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:04.360579 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:03.755941 1111 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.361112 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:03.959384 618 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:04.332631 30994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.303s user 0.002s sys 0.021s
I20250902 01:59:03.273399 1000 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:03.991775 1135 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.928097 974 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.364652 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.365453 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.365490 31695 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:04.007769 1137 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.218618 1002 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:03.995563 1032 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:03.852602 1034 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:03.998071 1128 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.595553 1034 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.392864 30965 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:04.595818 1128 consensus_queue.cc:1046] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250902 01:59:04.442008 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 85594.6us
I20250902 01:59:04.597143 30889 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:04.598299 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:04.392366 31067 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:04.391168 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.600147 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.600478 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.601218 31000 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:04.316831 1143 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.602658 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:04.368898 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:04.603488 31730 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:04.603943 1143 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:04.452713 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 437037us
I20250902 01:59:04.601233 992 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:04.605528 992 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:04.605932 992 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:03.757484 1072 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.607331 1072 raft_consensus.cc:3053] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:04.607667 1072 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:04.608386 1072 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.474900 30999 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:03.927627 1123 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.269677 1134 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:04.342984 1004 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:04.484361 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.480929 1026 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.544063 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:03.964388 789 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:04.356053 1051 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:04.562386 31070 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:04.390761 1084 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:04.359321 1006 consensus_queue.cc:1046] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.183s
I20250902 01:59:04.570165 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1919) took 3274 ms. Trace:
I20250902 01:59:04.627079 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.484472 987 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.392336 876 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:04.362442 1063 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:04.610867 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.551255 1020 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:04.612275 30819 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:04.612457 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 219708us
W20250902 01:59:04.615942 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1840) took 3271 ms (client timeout 1743 ms). Trace:
I20250902 01:59:04.621526 1143 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:04.616570 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.642058 30996 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:04.562862 1124 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:04.617446 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 252073us
I20250902 01:59:04.617974 30992 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:04.363646 1145 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.629146 31724 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250902 01:59:04.574190 700 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:04.546072 996 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.611939 1132 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.203913 1114 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.481633 899 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.610499 1123 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.566018 1091 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.617162 1111 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.616142 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.637940 30819 catalog_manager.cc:5582] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
W20250902 01:59:04.637917 30923 rpcz_store.cc:269] 0902 01:59:01.344837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.344980 (+ 143us) service_pool.cc:225] Handling call
0902 01:59:04.615763 (+3270783us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540013fe54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:04.615875 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":464789760}
I20250902 01:59:04.626619 789 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.206110 1079 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.644438 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:04.651280 347 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.625663 1119 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.486370 622 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.627664 1094 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.624197 1059 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.592478 1000 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.641198 876 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:04.646770 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.742486 31072 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:59:04.626442 535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:51240 (request call id 592) took 4592 ms. Trace:
I20250902 01:59:04.649881 31070 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:04.651574 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:04.612656 974 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.626978 30994 rpcz_store.cc:276] 0902 01:59:01.295996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.296135 (+ 139us) service_pool.cc:225] Handling call
0902 01:59:01.319067 (+ 22932us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:01.319244 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:03.029770 (+1710526us) log.cc:844] Serialized 14 byte log entry
0902 01:59:04.569876 (+1540106us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:04.569921 (+ 45us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:04.569961 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:04.570020 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:04.570120 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3261,"prepare.queue_time_us":1095131,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":44,"raft.queue_time_us":62,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":16321,"thread_start_us":3249937,"threads_started":2,"wal-append.queue_time_us":1414045}
I20250902 01:59:04.611267 1002 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.621089 1147 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:04.441870 1152 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.280s
I20250902 01:59:04.657316 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.680585 949 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.570570 1027 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.646435 1124 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:04.670584 905 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.633739 1084 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.676962 30965 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:04.765163 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 88286us
I20250902 01:59:04.589326 981 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:04.678750 1121 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 2.004s user 0.005s sys 0.018s
I20250902 01:59:04.655682 1060 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.683123 694 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.682665 1110 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.684288 1047 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:04.774207 1047 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.774780 620 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.651450 1051 consensus_queue.cc:1035] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250902 01:59:04.775311 620 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.649605 1020 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.777585 30997 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:04.689563 30917 raft_consensus.cc:3058] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:04.694814 30819 catalog_manager.cc:5582] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:04.653095 987 consensus_queue.cc:1046] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.027s
I20250902 01:59:04.743634 30892 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:04.743852 31069 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:04.745301 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:04.656055 700 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.676872 1058 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.749105 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:04.681142 1137 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:04.787616 31074 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:04.751701 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:04.368011 1109 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.757306 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:04.756165 30992 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:04.788239 31070 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:04.758797 31000 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:04.689280 1001 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.759107 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.789634 30994 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:04.790232 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:04.653470 1004 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.674865 1111 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.762763 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.763367 31073 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:04.674021 1063 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.622162 1134 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.689494 1123 consensus_queue.cc:1035] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20250902 01:59:04.767086 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:04.767956 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 200921us
I20250902 01:59:04.770221 1103 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.771147 1013 raft_consensus.cc:3053] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:59:04.688207 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 82321us
I20250902 01:59:04.621177 1135 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:04.689195 1143 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20250902 01:59:04.758308 1085 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.786212 31067 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:04.661536 1132 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.765316 912 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.686257 1126 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.791684 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:04.771876 1149 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:04.772091 905 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:04.779524 1156 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.795431 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:04.817903 1163 log.cc:927] Time spent T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20250902 01:59:04.797138 30819 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 4 to 5, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 5 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:04.784408 694 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.868460 31724 raft_consensus.cc:1273] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250902 01:59:04.817410 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:04.837983 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:04.800971 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.839134 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:04.864805 31038 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:04.863265 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 96323.8us
I20250902 01:59:04.869086 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:04.784307 1060 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:04.837666 1142 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20250902 01:59:04.652029 1145 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.919554 620 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:04.995990 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1921) took 3699 ms. Trace:
I20250902 01:59:04.819407 1004 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:04.796667 1024 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:04.758100 1147 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:04.793780 1162 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:05.086498 31069 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:04.785192 1077 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250902 01:59:04.986333 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:05.095188 31070 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:05.039924 1047 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:05.039844 1170 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:04.863739 1013 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:05.143668 30965 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:05.142532 30819 catalog_manager.cc:5582] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:59:04.817914 1069 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:04.817853 1063 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:04.997160 1111 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.999374 31728 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:04.912104 1110 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.281s
I20250902 01:59:04.796612 1167 consensus_queue.cc:1035] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.632s
W20250902 01:59:05.156020 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 291297us
I20250902 01:59:05.551465 1135 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:04.865499 1053 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:05.159569 1156 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:05.906203 31038 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:05.906675 1156 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:05.907191 1135 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:05.907755 1135 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:05.908843 1135 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:05.907665 30998 rpcz_store.cc:276] 0902 01:59:01.296655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.405973 (+109318us) service_pool.cc:225] Handling call
0902 01:59:01.536351 (+130378us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:02.921625 (+1385274us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:02.921655 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:02.926730 (+ 5075us) log.cc:844] Serialized 26 byte log entry
0902 01:59:03.312719 (+385989us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:03.312773 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:04.872936 (+1560163us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001af9d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:04.872993 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:04.873061 (+ 68us) raft_consensus.cc:1683] finished
0902 01:59:04.873074 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:04.995913 (+122839us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5177,"raft.queue_time_us":5610,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":9559,"spinlock_wait_cycles":3173364992,"thread_start_us":385671,"threads_started":1,"wal-append.queue_time_us":1037805,"wal-append.run_cpu_time_us":6693,"wal-append.run_wall_time_us":1084958}
I20250902 01:59:05.907294 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1737) took 1936 ms. Trace:
I20250902 01:59:05.909863 31070 rpcz_store.cc:276] 0902 01:59:03.970995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.751402 (+780407us) service_pool.cc:225] Handling call
0902 01:59:05.907239 (+1155837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:05.084589 1091 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:05.910061 1024 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:05.908278 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:04.746258 535 rpcz_store.cc:276] 0902 01:59:00.031828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.038498 (+ 6670us) service_pool.cc:225] Handling call
0902 01:59:00.377908 (+339410us) ts_tablet_manager.cc:928] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Initiating tablet copy from peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457)
0902 01:59:03.791268 (+3413360us) tablet_copy_client.cc:368] Tablet copy session begun
0902 01:59:03.802862 (+ 11594us) tablet_metadata.cc:641] Metadata flushed
0902 01:59:03.803189 (+ 327us) tablet_copy_client.cc:497] Wrote new tablet metadata
0902 01:59:04.335828 (+532639us) tablet_copy_client.cc:418] Wrote new consensus metadata
0902 01:59:04.479069 (+143241us) tablet_replica.cc:206] Creating consensus instance
0902 01:59:04.588892 (+109823us) ts_tablet_manager.cc:983] Replica registered.
0902 01:59:04.611103 (+ 22211us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":179,"tablet-copy.queue_time_us":301378,"thread_start_us":382694,"threads_started":2}
W20250902 01:59:05.826099 30917 consensus_meta.cc:287] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.048s user 0.004s sys 0.000s
I20250902 01:59:05.911656 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:05.976836 30917 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:05.977445 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1763) took 1195 ms. Trace:
I20250902 01:59:05.978515 31070 rpcz_store.cc:276] 0902 01:59:04.781886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.911428 (+1129542us) service_pool.cc:225] Handling call
0902 01:59:05.977372 (+ 65944us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:05.901381 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:05.776036 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:05.948205 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:05.269697 1146 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
W20250902 01:59:05.984310 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.332s user 0.020s sys 0.010s
W20250902 01:59:05.984606 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.020s sys 0.010s
W20250902 01:59:05.839289 1124 consensus_meta.cc:287] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.067s user 0.000s sys 0.007s
W20250902 01:59:05.572351 31000 consensus_meta.cc:287] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.783s user 0.008s sys 0.000s
I20250902 01:59:05.987840 31000 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:05.912884 1147 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:05.988529 1147 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:05.989080 1147 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:05.990677 1109 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:05.991250 1147 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:05.271090 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.034s sys 0.051s
I20250902 01:59:05.972023 1053 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:05.992056 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.630s user 0.041s sys 0.065s
I20250902 01:59:05.992319 1053 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:05.992712 1053 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:05.253602 1114 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:05.193730 618 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:05.964578 1079 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:05.995911 618 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:05.987048 1047 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:05.996290 618 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:05.996464 1047 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:05.998330 1047 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:05.172008 1128 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:05.811093 1134 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:05.969899 1020 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:05.268435 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:05.193760 1158 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.519s user 0.000s sys 0.001s
I20250902 01:59:05.903015 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1839) took 2091 ms. Trace:
W20250902 01:59:06.002705 31067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1748) took 2007 ms (client timeout 2216 ms). Trace:
W20250902 01:59:05.901904 30999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1924) took 4583 ms (client timeout 1624 ms). Trace:
I20250902 01:59:05.590358 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:05.973107 30819 catalog_manager.cc:5582] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 2 to 3, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 088424002a894fd5bcc11a06af111445 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:05.973263 1024 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:05.980712 30992 raft_consensus.cc:2408] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250902 01:59:05.980706 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 893) took 1201 ms (client timeout 1579 ms). Trace:
W20250902 01:59:05.988757 992 consensus_meta.cc:287] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.382s user 0.009s sys 0.008s
I20250902 01:59:05.981451 31730 raft_consensus.cc:2366] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
W20250902 01:59:05.269889 1153 log.cc:927] Time spent T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20250902 01:59:05.982698 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 614) took 1107 ms. Trace:
I20250902 01:59:06.020359 31727 rpcz_store.cc:276] 0902 01:59:04.872628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.040911 (+168283us) service_pool.cc:225] Handling call
0902 01:59:05.979990 (+939079us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.021219 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.021631 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.021948 30996 raft_consensus.cc:2391] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:06.022325 31727 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:05.119226 899 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:06.024022 1145 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.024147 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.025076 31727 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:06.026190 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.027120 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 615) took 1117 ms. Trace:
I20250902 01:59:06.027594 31727 rpcz_store.cc:276] 0902 01:59:04.909248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.025901 (+1116653us) service_pool.cc:225] Handling call
0902 01:59:06.027060 (+ 1159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.029364 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:05.903092 694 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:05.991793 1170 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:05.992058 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.030717 31723 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:06.031356 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 887) took 1277 ms. Trace:
I20250902 01:59:06.031762 1135 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.031869 1170 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:06.031822 31723 rpcz_store.cc:276] 0902 01:59:04.753534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.991765 (+1238231us) service_pool.cc:225] Handling call
0902 01:59:06.031286 (+ 39521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.033680 1135 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:06.030712 31724 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:06.036405 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 602) took 1286 ms. Trace:
I20250902 01:59:06.036887 31724 rpcz_store.cc:276] 0902 01:59:04.749585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.493517 (+743932us) service_pool.cc:225] Handling call
0902 01:59:06.036179 (+542662us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b54005ea954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:06.036328 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52019456}
I20250902 01:59:05.040069 905 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.000448 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.000429 31072 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
I20250902 01:59:06.039808 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1764) took 1249 ms. Trace:
I20250902 01:59:06.040108 31072 rpcz_store.cc:276] 0902 01:59:04.789982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.171382 (+381400us) service_pool.cc:225] Handling call
0902 01:59:06.039734 (+868352us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.040628 905 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.040884 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.041293 31729 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:06.041433 31072 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:06.041956 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1765) took 1248 ms. Trace:
I20250902 01:59:06.042236 31072 rpcz_store.cc:276] 0902 01:59:04.793733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.040647 (+1246914us) service_pool.cc:225] Handling call
0902 01:59:06.041907 (+ 1260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.043038 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:05.369035 1149 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:06.043794 1149 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.045805 1149 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:06.000846 31692 connection.cc:232] RPC call timeout handler was delayed by -1.5679992800000946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
I20250902 01:59:06.039283 31070 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:06.046741 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1761) took 1370 ms. Trace:
I20250902 01:59:06.047044 31070 rpcz_store.cc:276] 0902 01:59:04.676088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.986270 (+1310182us) service_pool.cc:225] Handling call
0902 01:59:06.046672 (+ 60402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.047928 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.048542 31070 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:06.049340 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.049989 31070 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:06.050946 31070 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250902 01:59:06.048554 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.052619 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 612) took 1254 ms. Trace:
W20250902 01:59:05.681037 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:05.371181 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.055624 30994 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 2.
W20250902 01:59:06.056298 30994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1945) took 1376 ms (client timeout 1778 ms). Trace:
W20250902 01:59:06.056731 30994 rpcz_store.cc:269] 0902 01:59:04.680579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.172371 (+491792us) service_pool.cc:225] Handling call
0902 01:59:06.056231 (+883860us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:05.953722 31069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1757) took 1625 ms (client timeout 1913 ms). Trace:
W20250902 01:59:06.058343 31069 rpcz_store.cc:269] 0902 01:59:04.328755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.640031 (+311276us) service_pool.cc:225] Handling call
0902 01:59:05.953656 (+1313625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.059722 31069 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:06.060411 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1739) took 2068 ms. Trace:
I20250902 01:59:06.060792 31069 rpcz_store.cc:276] 0902 01:59:03.992218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.059144 (+2066926us) service_pool.cc:225] Handling call
0902 01:59:06.059468 (+ 324us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.060160 (+ 692us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.060354 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.062763 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1742) took 2069 ms. Trace:
I20250902 01:59:06.064183 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:06.064944 30994 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:06.068776 30967 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.306s user 0.024s sys 0.074s
I20250902 01:59:05.432394 1000 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:05.972390 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 64194.8us
W20250902 01:59:06.003288 31067 rpcz_store.cc:269] 0902 01:59:03.994781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.599915 (+605134us) service_pool.cc:225] Handling call
0902 01:59:06.002555 (+1402640us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54001c3754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:06.002645 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426751360}
I20250902 01:59:06.069865 31727 rpcz_store.cc:276] 0902 01:59:04.849467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.029094 (+1179627us) service_pool.cc:225] Handling call
0902 01:59:06.052550 (+ 23456us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:06.010002 1106 log.cc:927] Time spent T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 3.540s user 0.024s sys 0.022s
I20250902 01:59:05.488281 974 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.004365 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 597) took 1258 ms. Trace:
I20250902 01:59:06.002552 30995 rpcz_store.cc:276] 0902 01:59:03.811679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.852510 (+ 40831us) service_pool.cc:225] Handling call
0902 01:59:03.927679 (+ 75169us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:04.675012 (+747333us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:04.675057 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:04.675116 (+ 59us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:04.675164 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:05.902786 (+1227622us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b5400178254. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:05.902833 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:05.902964 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1718307200}
W20250902 01:59:06.003515 30999 rpcz_store.cc:269] 0902 01:59:01.318334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.322010 (+ 3676us) service_pool.cc:225] Handling call
0902 01:59:05.901700 (+4579690us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54002ad5d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:05.901819 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4388148992}
I20250902 01:59:06.076787 31726 rpcz_store.cc:276] 0902 01:59:04.746965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.753009 (+ 6044us) service_pool.cc:225] Handling call
0902 01:59:04.762493 (+ 9484us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:04.777312 (+ 14819us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:04.777341 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:04.777383 (+ 42us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:04.777412 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.004161 (+1226749us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54004d53d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:06.004204 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.004315 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33698176}
I20250902 01:59:06.077476 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:05.356861 1132 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:05.950578 981 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:05.161998 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 292977us
I20250902 01:59:05.510057 1027 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.080734 31039 leader_election.cc:400] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:05.950793 1077 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:06.007974 31728 rpcz_store.cc:269] 0902 01:59:04.769285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.786345 (+ 17060us) service_pool.cc:225] Handling call
0902 01:59:05.970387 (+1184042us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:05.898368 1171 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.776s user 0.001s sys 0.000s
I20250902 01:59:05.980563 1162 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:06.016666 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 883) took 1272 ms. Trace:
I20250902 01:59:06.085834 31730 rpcz_store.cc:276] 0902 01:59:04.744611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.744719 (+ 108us) service_pool.cc:225] Handling call
0902 01:59:06.016608 (+1271889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.009232 30819 catalog_manager.cc:5582] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 reported cstate change: term changed from 3 to 4, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:05.984118 1172 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:06.005923 1167 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20250902 01:59:06.005793 1114 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20250902 01:59:06.018376 992 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:06.009193 30992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1937) took 1796 ms (client timeout 2241 ms). Trace:
I20250902 01:59:06.005559 1124 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.019748 1113 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:06.022377 30996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 399) took 1666 ms (client timeout 2008 ms). Trace:
W20250902 01:59:06.094054 30996 rpcz_store.cc:269] 0902 01:59:04.356407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.649583 (+293176us) service_pool.cc:225] Handling call
0902 01:59:06.021124 (+1371541us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff19166c 00007fc5ff1915d6 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:06.022325 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405559296}
I20250902 01:59:06.085367 1053 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.095005 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.095738 30996 raft_consensus.cc:2408] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:06.095971 992 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.096386 1091 raft_consensus.cc:3053] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:06.096504 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.096843 1091 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:06.097213 30996 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
W20250902 01:59:05.396394 1164 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.649s user 0.000s sys 0.001s
W20250902 01:59:05.950151 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 806621us
I20250902 01:59:06.024708 1013 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.097993 1124 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.097726 1091 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.098500 1013 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:06.098716 30965 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:05.989482 535 tablet_copy_client.cc:806] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 0 data blocks...
I20250902 01:59:06.100188 535 tablet_copy_client.cc:670] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Starting download of 1 WAL segments...
I20250902 01:59:06.038321 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.101032 31723 raft_consensus.cc:2408] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:06.101426 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 611) took 1310 ms. Trace:
I20250902 01:59:06.087838 1172 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:06.101855 1172 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:06.102419 1000 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:06.103013 1000 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:06.086611 1162 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:06.104017 30924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1841) took 4758 ms (client timeout 1629 ms). Trace:
W20250902 01:59:06.104358 30924 rpcz_store.cc:269] 0902 01:59:01.345279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.570255 (+1224976us) service_pool.cc:225] Handling call
0902 01:59:06.103843 (+3533588us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540034f8d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:06.103949 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243416320}
W20250902 01:59:05.986744 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:06.105293 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:06.052894 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:04.799294 1141 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:06.098721 1091 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:06.114552 31074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1753) took 2117 ms (client timeout 1519 ms). Trace:
W20250902 01:59:06.114871 31074 rpcz_store.cc:269] 0902 01:59:03.997490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.210411 (+212921us) service_pool.cc:225] Handling call
0902 01:59:06.114384 (+1903973us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54001eafd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:06.114468 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1325560704}
I20250902 01:59:06.115869 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.054032 31729 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:06.116251 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1744) took 2122 ms. Trace:
W20250902 01:59:06.119087 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 881) took 5359 ms (client timeout 1629 ms). Trace:
I20250902 01:59:06.166250 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.180615 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:06.069248 30967 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.306s user 0.025s sys 0.074s
I20250902 01:59:05.957970 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:06.325376 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 1.56239e+06us
W20250902 01:59:06.123088 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:06.325811 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 367949us
I20250902 01:59:06.320712 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.059234 618 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:59:06.076103 31692 connection.cc:232] RPC call timeout handler was delayed by -1.5330895499999997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2088s for a response
I20250902 01:59:06.076177 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.078751 30994 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:06.079304 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1844) took 1699 ms. Trace:
I20250902 01:59:05.537567 876 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.081027 31039 leader_election.cc:403] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20250902 01:59:06.324218 31725 rpcz_store.cc:269] 0902 01:59:00.759683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.773697 (+ 14014us) service_pool.cc:225] Handling call
0902 01:59:02.744999 (+1971302us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:06.118915 (+3373916us) spinlock_profiling.cc:243] Waited 65.2 ms on lock 0x7b5400740754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:06.118995 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273468032}
I20250902 01:59:06.081382 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:06.081670 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:06.072095 1112 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.020s sys 0.023s
I20250902 01:59:06.086642 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.087013 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.088032 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.084972 1077 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:59:06.093035 30992 rpcz_store.cc:269] 0902 01:59:04.213026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.171131 (+958105us) service_pool.cc:225] Handling call
0902 01:59:06.009139 (+838008us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24320}
I20250902 01:59:06.096125 1053 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.101702 31723 rpcz_store.cc:276] 0902 01:59:04.791144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.037991 (+1246847us) service_pool.cc:225] Handling call
0902 01:59:06.101376 (+ 63385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:05.548025 1084 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.051406 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1738) took 2059 ms. Trace:
I20250902 01:59:06.098023 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.179068 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.326519 30892 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:06.070837 31067 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20250902 01:59:06.327001 30889 connection.cc:232] RPC call timeout handler was delayed by -1.0306973090000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22410000000000002s for a response
I20250902 01:59:06.328502 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.329095 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.063174 31069 rpcz_store.cc:276] 0902 01:59:03.993137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.061735 (+2068598us) service_pool.cc:225] Handling call
0902 01:59:06.062152 (+ 417us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.062357 (+ 205us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.062393 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:06.062448 (+ 55us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:06.062501 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.062567 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.062704 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.116822 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.324052 31074 rpcz_store.cc:276] 0902 01:59:03.993660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.115580 (+2121920us) service_pool.cc:225] Handling call
0902 01:59:06.115833 (+ 253us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.115989 (+ 156us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.116022 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:06.116055 (+ 33us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:06.116081 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.116125 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.116210 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.684178 31070 rpcz_store.cc:276] 0902 01:59:03.991883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.050442 (+2058559us) service_pool.cc:225] Handling call
0902 01:59:06.050679 (+ 237us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.051204 (+ 525us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.051363 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.327566 30924 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:06.516716 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:06.633406 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:06.685660 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 601) took 1936 ms. Trace:
I20250902 01:59:06.685742 30925 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:06.411970 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 616) took 1370 ms. Trace:
I20250902 01:59:06.684075 31728 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:06.331944 30999 rpcz_store.cc:276] 0902 01:59:04.380165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.077248 (+1697083us) service_pool.cc:225] Handling call
0902 01:59:06.079247 (+ 1999us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:06.507071 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 90850.9us
I20250902 01:59:06.332720 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:06.689231 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:06.686033 31726 rpcz_store.cc:276] 0902 01:59:04.749331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.081936 (+1332605us) service_pool.cc:225] Handling call
0902 01:59:06.685599 (+603663us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:06.520251 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:06.689716 30998 raft_consensus.cc:2408] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250902 01:59:06.686447 30889 connection.cc:232] RPC call timeout handler was delayed by -1.0306973090000611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22410000000000002s for a response
W20250902 01:59:06.691854 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 365409us
W20250902 01:59:06.691145 30994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 400) took 2331 ms (client timeout 2789 ms). Trace:
I20250902 01:59:06.692584 30892 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:06.450290 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 369636us
I20250902 01:59:06.693980 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:06.501045 31072 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:06.692409 30994 rpcz_store.cc:269] 0902 01:59:04.360211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.063880 (+1703669us) service_pool.cc:225] Handling call
0902 01:59:06.691096 (+627216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.694972 31727 rpcz_store.cc:276] 0902 01:59:05.041599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.075740 (+1034141us) service_pool.cc:225] Handling call
0902 01:59:06.411886 (+336146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.696319 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.697263 30992 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
W20250902 01:59:06.697729 30998 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1946) took 2016 ms (client timeout 2040 ms). Trace:
W20250902 01:59:06.696590 31075 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1631) took 2790 ms (client timeout 1840 ms). Trace:
I20250902 01:59:06.684618 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1743) took 2691 ms. Trace:
W20250902 01:59:06.689703 31038 connection.cc:232] RPC call timeout handler was delayed by -1.4198466789999884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
I20250902 01:59:06.689249 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 619) took 1507 ms. Trace:
I20250902 01:59:06.701004 31728 rpcz_store.cc:276] 0902 01:59:05.182187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.084069 (+901882us) service_pool.cc:225] Handling call
0902 01:59:06.689189 (+605120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:06.698084 30998 rpcz_store.cc:269] 0902 01:59:04.680867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:05.909654 (+1228787us) service_pool.cc:225] Handling call
0902 01:59:06.328376 (+418722us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b5800193898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:06.697675 (+369299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97032960}
I20250902 01:59:06.688402 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.699519 31067 rpcz_store.cc:276] 0902 01:59:03.993417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.070326 (+2076909us) service_pool.cc:225] Handling call
0902 01:59:06.070608 (+ 282us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:06.684414 (+613806us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.684578 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.702746 30999 raft_consensus.cc:2408] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250902 01:59:06.465252 31026 heartbeater.cc:646] Failed to heartbeat to 127.29.21.254:45755 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.21.254:45755 timed out after 15.000s (SENT)
I20250902 01:59:06.237442 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1627) took 5232 ms. Trace:
I20250902 01:59:06.175782 1174 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:06.707695 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.707984 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.707841 1174 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.708887 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.709065 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:06.709586 30924 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:06.709903 1174 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:06.709945 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.703871 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.323016 1072 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:06.117063 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 613) took 1249 ms. Trace:
I20250902 01:59:06.705049 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.699908 31039 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:06.702226 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.720894 30994 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:06.721206 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:06.706840 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:06.698366 31075 rpcz_store.cc:269] 0902 01:59:03.896791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.898139 (+ 1348us) service_pool.cc:225] Handling call
0902 01:59:06.328995 (+2430856us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:06.686993 (+357998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23314176}
W20250902 01:59:06.502221 1173 log.cc:927] Time spent T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.424s user 0.000s sys 0.000s
I20250902 01:59:06.711334 31725 raft_consensus.cc:2408] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:06.502058 618 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:06.685307 1067 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.871s user 0.026s sys 0.051s
W20250902 01:59:06.414232 1112 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s user 0.020s sys 0.023s
I20250902 01:59:06.713224 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.714144 30992 raft_consensus.cc:2391] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:06.704396 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.730811 31727 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:06.730967 30996 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:06.731400 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 401) took 2368 ms. Trace:
I20250902 01:59:06.118842 1026 raft_consensus.cc:3053] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:06.711273 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.714787 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.705205 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1745) took 2711 ms. Trace:
I20250902 01:59:06.652292 1167 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:06.733618 31728 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:06.498924 1172 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.495080 1162 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.734112 31074 rpcz_store.cc:276] 0902 01:59:03.993940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.698839 (+2704899us) service_pool.cc:225] Handling call
0902 01:59:06.704545 (+ 5706us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.704730 (+ 185us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.704751 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:06.704795 (+ 44us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:06.704824 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.704875 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.705138 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.124886 622 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:06.413489 1124 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:06.471793 1027 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:06.716841 31692 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:06.413743 990 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:06.462915 1091 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:06.706102 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.750398 31730 raft_consensus.cc:2374] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
I20250902 01:59:06.751195 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.722056 30998 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:06.719030 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 481363us
I20250902 01:59:06.751925 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.752588 31726 raft_consensus.cc:2366] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:06.722107 30999 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:06.753286 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.753573 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.753815 31726 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:06.753772 31692 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:06.754649 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.754901 31692 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:06.755578 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.756088 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:06.756278 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.756855 30994 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:06.722322 31067 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:06.723255 30925 raft_consensus.cc:2391] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:06.185859 1097 log.cc:927] Time spent T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 3.163s user 0.006s sys 0.071s
I20250902 01:59:06.471729 1114 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:06.723620 31693 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:06.723809 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.332130 1122 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:06.731527 618 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:06.703680 1069 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:06.703761 991 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:06.710693 1084 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:06.731737 30996 rpcz_store.cc:276] 0902 01:59:04.363239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.697045 (+2333806us) service_pool.cc:225] Handling call
0902 01:59:06.731352 (+ 34307us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.500571 1024 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.770704 1084 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:06.771060 1084 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:06.771371 1084 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:06.771930 1024 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.772511 1024 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:06.735209 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.773242 30922 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:06.774003 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.735625 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 892) took 1966 ms. Trace:
I20250902 01:59:06.774784 31728 rpcz_store.cc:276] 0902 01:59:04.768989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.711042 (+1942053us) service_pool.cc:225] Handling call
0902 01:59:06.735516 (+ 24474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.765513 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.775645 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.777120 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.778079 30996 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:06.332458 992 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:06.736150 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.731662 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.780589 31727 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:06.781517 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.703575 876 consensus_queue.cc:1046] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20250902 01:59:06.740428 31074 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:06.782936 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1842) took 2417 ms. Trace:
I20250902 01:59:06.720867 31724 rpcz_store.cc:276] 0902 01:59:04.867950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.052645 (+1184695us) service_pool.cc:225] Handling call
0902 01:59:06.117001 (+ 64356us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.743408 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 875) took 6502 ms. Trace:
I20250902 01:59:06.732955 1026 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:06.739584 1167 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:06.731432 1067 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.916s user 0.026s sys 0.052s
I20250902 01:59:06.757836 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:06.759099 30999 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:59:06.729804 30921 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.002s sys 0.023s
W20250902 01:59:06.804009 30921 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.685s user 0.002s sys 0.023s
W20250902 01:59:06.805665 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 47881.7us
I20250902 01:59:06.806531 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:06.762172 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1755) took 2764 ms. Trace:
I20250902 01:59:06.816238 31067 rpcz_store.cc:276] 0902 01:59:03.997952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.703161 (+2705209us) service_pool.cc:225] Handling call
0902 01:59:06.721800 (+ 18639us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.762012 (+ 40212us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.762132 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.764990 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.766868 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.774485 30922 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:06.779409 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:06.818630 31728 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:06.783260 30996 rpcz_store.cc:276] 0902 01:59:04.365985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.767145 (+2401160us) service_pool.cc:225] Handling call
0902 01:59:06.782888 (+ 15743us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.784168 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:06.752066 1027 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.784585 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.761072 620 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:06.768111 1113 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:06.788341 31723 raft_consensus.cc:2366] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4.
I20250902 01:59:06.820887 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.825119 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:06.826435 31730 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:06.826633 31728 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:06.779016 1069 raft_consensus.cc:695] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:06.758061 990 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.824016 620 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:06.789716 1139 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:06.762667 1114 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.764199 1122 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.757071 1124 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:06.827193 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 889) took 2059 ms. Trace:
I20250902 01:59:06.768203 618 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:06.803144 1174 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:06.841079 30992 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:06.827553 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:06.828804 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.829548 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1932) took 2617 ms. Trace:
I20250902 01:59:06.830350 31730 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "349593d2d3574cd6ab281c105756f435"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:59:06.820107 1024 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:06.820439 992 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:06.778888 991 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.852185 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.861871 30923 raft_consensus.cc:2408] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:06.853641 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.862700 30922 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:06.854333 30994 rpcz_store.cc:276] 0902 01:59:04.211768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.820252 (+2608484us) service_pool.cc:225] Handling call
0902 01:59:06.829066 (+ 8814us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.829252 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.829274 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:06.829333 (+ 59us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:06.829365 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.829413 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.829500 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.839385 31066 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:06.843511 31728 rpcz_store.cc:276] 0902 01:59:04.768125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.824830 (+2056705us) service_pool.cc:225] Handling call
0902 01:59:06.827129 (+ 2299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:06.854193 30925 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:06.851481 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:06.972946 31074 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:06.823781 1172 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.974123 30995 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:06.974668 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:06.839648 1063 consensus_meta.cc:287] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.859s user 0.004s sys 0.004s
I20250902 01:59:06.979957 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:07.029096 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 336585us
I20250902 01:59:06.978440 31728 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250902 01:59:07.031081 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 886) took 2279 ms. Trace:
I20250902 01:59:07.031539 31728 rpcz_store.cc:276] 0902 01:59:04.752069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.977908 (+2225839us) service_pool.cc:225] Handling call
0902 01:59:06.978175 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.030797 (+ 52622us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.031027 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:07.033861 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 603) took 2281 ms. Trace:
I20250902 01:59:07.038475 31728 rpcz_store.cc:276] 0902 01:59:04.752526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.032850 (+2280324us) service_pool.cc:225] Handling call
0902 01:59:07.033172 (+ 322us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.033482 (+ 310us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:07.033526 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.033593 (+ 67us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:07.033628 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.033691 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.033811 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:07.042284 30918 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:07.030308 30892 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:06.833159 1162 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.836575 1027 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:06.835106 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:07.043154 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.834955 1114 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:06.831934 1139 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:07.060551 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:07.056422 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:07.061368 31074 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:07.045663 30892 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:07.073380 30892 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:07.080018 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:07.108430 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:06.830121 620 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:07.053169 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:07.110342 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 410559us
I20250902 01:59:07.047495 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 604) took 2293 ms. Trace:
W20250902 01:59:07.109588 31073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1756) took 3103 ms (client timeout 2169 ms). Trace:
W20250902 01:59:07.111321 31073 rpcz_store.cc:269] 0902 01:59:04.006177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.612989 (+606812us) service_pool.cc:225] Handling call
0902 01:59:07.109440 (+2496451us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b540024c7d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.109522 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211336064}
I20250902 01:59:07.111186 31728 rpcz_store.cc:276] 0902 01:59:04.753873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.046634 (+2292761us) service_pool.cc:225] Handling call
0902 01:59:07.046959 (+ 325us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.047145 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:07.047180 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.047233 (+ 53us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:07.047271 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.047335 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.047441 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:07.112367 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 356348us
I20250902 01:59:07.112957 31692 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:07.111071 31039 leader_election.cc:400] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:07.113488 31039 leader_election.cc:403] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250902 01:59:07.113580 31692 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:07.114462 31692 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:07.053812 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1933) took 2841 ms. Trace:
I20250902 01:59:07.115548 30994 rpcz_store.cc:276] 0902 01:59:04.212059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.978561 (+2766502us) service_pool.cc:225] Handling call
0902 01:59:07.053264 (+ 74703us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.053462 (+ 198us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:07.053494 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.053543 (+ 49us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:07.053581 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.053655 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.053757 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:07.118316 31039 connection.cc:232] RPC call timeout handler was delayed by -4.5312870090000388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
W20250902 01:59:07.132843 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:06.857795 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 166647us
W20250902 01:59:07.151422 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 427907us
I20250902 01:59:06.829680 1113 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:06.853524 1174 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:06.861233 992 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.164925 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:07.165550 992 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:07.166440 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:07.164966 1192 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:07.168186 1192 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:07.168275 990 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.168547 1192 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:07.169133 535 tablet_copy_client.cc:538] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250902 01:59:07.170658 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:07.152463 30967 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:07.171202 31073 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:07.171710 30967 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:06.857336 1024 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.155375 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:07.059476 1139 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:06.787357 31721 rpcz_store.cc:276] 0902 01:59:00.241188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:00.771363 (+530175us) service_pool.cc:225] Handling call
0902 01:59:02.744092 (+1972729us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:59:02.744335 (+ 243us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:04.205259 (+1460924us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:06.743166 (+2537907us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54006ae0d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:06.743335 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":906818304}
I20250902 01:59:07.028551 1026 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:07.162169 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1934) took 2949 ms. Trace:
I20250902 01:59:07.156067 31693 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:06.828053 1069 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:06.707475 31071 rpcz_store.cc:276] 0902 01:59:01.002876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:01.003014 (+ 138us) service_pool.cc:225] Handling call
0902 01:59:01.003469 (+ 455us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:01.003850 (+ 381us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:01.003880 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:01.004378 (+ 498us) log.cc:844] Serialized 26 byte log entry
0902 01:59:02.728370 (+1723992us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:02.728414 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:02.728481 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:05.982950 (+3254469us) spinlock_profiling.cc:243] Waited 30.8 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:06.175813 (+192863us) raft_consensus.cc:1683] finished
0902 01:59:06.175836 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:06.234122 (+ 58286us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1226,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":70761344,"thread_start_us":1723571,"threads_started":1,"wal-append.queue_time_us":1465246}
I20250902 01:59:07.154932 1132 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457) with higher term. Message: Invalid argument: T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:07.164745 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 902) took 1077 ms. Trace:
I20250902 01:59:07.165015 1174 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:07.392938 31075 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.003s sys 0.028s
W20250902 01:59:07.543123 30998 consensus_meta.cc:287] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.791s user 0.002s sys 0.003s
I20250902 01:59:07.554322 618 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:07.329864 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1750) took 3334 ms. Trace:
I20250902 01:59:07.150560 1148 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:07.323946 1130 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:07.565008 31070 rpcz_store.cc:276] 0902 01:59:03.995294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.700781 (+2705487us) service_pool.cc:225] Handling call
0902 01:59:06.706593 (+ 5812us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.774743 (+ 68150us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.774783 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:06.774832 (+ 49us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:06.774871 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.329609 (+554738us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b54001e82d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:07.329664 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.329801 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156244864}
W20250902 01:59:07.565655 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 492406us
I20250902 01:59:07.372505 1141 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:07.328313 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:07.577390 1112 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.017s sys 0.033s
I20250902 01:59:07.441402 522 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:07.168298 1124 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:07.530267 1192 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:07.589133 30997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1941) took 3252 ms (client timeout 2015 ms). Trace:
W20250902 01:59:07.589493 30997 rpcz_store.cc:269] 0902 01:59:04.336422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.360379 (+ 23957us) service_pool.cc:225] Handling call
0902 01:59:07.588993 (+3228614us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400179654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.589078 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":495185152}
I20250902 01:59:07.589686 620 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:07.590484 1192 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:07.590672 31728 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:07.567678 1024 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:07.370690 1122 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.232388 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:07.591459 1065 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.324159 993 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:59:07.591777 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 605) took 2837 ms. Trace:
I20250902 01:59:07.591933 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1926) took 6215 ms. Trace:
I20250902 01:59:07.200767 1191 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:07.592141 993 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:07.485471 1077 consensus_meta.cc:287] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.771s user 0.010s sys 0.000s
I20250902 01:59:07.592700 993 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:07.564241 30994 rpcz_store.cc:276] 0902 01:59:04.212310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.154821 (+2942511us) service_pool.cc:225] Handling call
0902 01:59:07.161612 (+ 6791us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.161821 (+ 209us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:07.161860 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.161909 (+ 49us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:07.161948 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.162009 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.162114 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:07.390205 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1836) took 3579 ms. Trace:
I20250902 01:59:07.312583 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:07.338796 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 224384us
I20250902 01:59:07.570910 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:07.472847 992 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:07.306515 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1930) took 3095 ms. Trace:
W20250902 01:59:07.564291 31075 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.004s sys 0.038s
I20250902 01:59:07.305914 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 484) took 1199 ms. Trace:
I20250902 01:59:07.572526 700 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.372939 1046 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.245s
I20250902 01:59:07.572376 31724 rpcz_store.cc:276] 0902 01:59:06.087300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.809377 (+722077us) service_pool.cc:225] Handling call
0902 01:59:07.164684 (+355307us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:07.475492 997 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:07.436062 1185 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:07.191359 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:07.440272 991 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:07.396536 1084 consensus_meta.cc:287] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.625s user 0.006s sys 0.002s
I20250902 01:59:07.576035 31074 raft_consensus.cc:2391] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:07.437892 1054 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:07.576664 1067 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.006s sys 0.017s
I20250902 01:59:07.078126 1027 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:07.231945 31039 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.29.21.254:45755, user_credentials={real_user=slave}} blocked reactor thread for 45120.3us
I20250902 01:59:07.495714 1113 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:07.580719 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 118161us
I20250902 01:59:07.580715 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:07.577903 622 consensus_meta.cc:287] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.828s user 0.009s sys 0.000s
W20250902 01:59:07.584860 31066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1754) took 3587 ms (client timeout 2015 ms). Trace:
W20250902 01:59:07.586267 31725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 626) took 1431 ms (client timeout 1516 ms). Trace:
I20250902 01:59:07.555648 1114 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:07.580179 1148 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.582033 1141 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:07.577662 30891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.001s sys 0.012s
W20250902 01:59:07.461589 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.005s sys 0.013s
W20250902 01:59:07.348896 30924 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.639s user 0.009s sys 0.000s
I20250902 01:59:07.302589 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 485) took 1186 ms. Trace:
I20250902 01:59:07.171211 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:07.585572 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 405) took 1589 ms. Trace:
W20250902 01:59:07.575013 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 403473us
W20250902 01:59:07.587136 30995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1939) took 3251 ms (client timeout 2934 ms). Trace:
W20250902 01:59:07.468432 30996 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.005s sys 0.012s
I20250902 01:59:07.564456 30998 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:59:07.587932 31000 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1847) took 3045 ms (client timeout 1725 ms). Trace:
I20250902 01:59:07.679031 535 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap starting.
W20250902 01:59:08.054378 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:08.238032 30964 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.763s (SENT) [suppressed 37 similar messages]
I20250902 01:59:07.683773 1002 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:07.860363 620 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:08.321260 1189 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.838s user 0.000s sys 0.007s
W20250902 01:59:07.680164 31072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1751) took 3684 ms (client timeout 2934 ms). Trace:
I20250902 01:59:08.322197 30924 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
W20250902 01:59:08.322367 31072 rpcz_store.cc:269] 0902 01:59:03.996949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.042794 (+2045845us) service_pool.cc:225] Handling call
0902 01:59:07.679951 (+1637157us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b540036db54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.680096 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1051852544}
I20250902 01:59:07.709731 1060 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:08.323807 31000 rpcz_store.cc:269] 0902 01:59:04.542849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.601875 (+ 59026us) service_pool.cc:225] Handling call
0902 01:59:07.587743 (+2985868us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400127354. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.587868 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":357823360}
W20250902 01:59:08.324373 1207 log.cc:927] Time spent T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.756s user 0.001s sys 0.008s
W20250902 01:59:07.672722 31729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 885) took 2926 ms (client timeout 1512 ms). Trace:
I20250902 01:59:08.321172 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1760) took 3664 ms. Trace:
W20250902 01:59:08.323293 30924 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1674) took 1621 ms (client timeout 1846 ms). Trace:
I20250902 01:59:08.099109 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1633) took 4143 ms. Trace:
W20250902 01:59:08.318401 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 80575.5us
W20250902 01:59:07.582691 1112 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.833s user 0.017s sys 0.034s
W20250902 01:59:08.100407 30917 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1845) took 6735 ms (client timeout 1706 ms). Trace:
I20250902 01:59:08.176798 789 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.575320 1204 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:07.208235 876 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:08.224691 832 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:07.681811 1196 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.495s user 0.002s sys 0.016s
I20250902 01:59:08.061401 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:08.097661 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:08.104703 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:08.327289 30924 rpcz_store.cc:269] 0902 01:59:06.701821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.708853 (+ 7032us) service_pool.cc:225] Handling call
0902 01:59:08.323160 (+1614307us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:08.115602 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 959484us
W20250902 01:59:08.326292 31729 rpcz_store.cc:269] 0902 01:59:04.746270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.746398 (+ 128us) service_pool.cc:225] Handling call
0902 01:59:07.672515 (+2926117us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54004d0154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.672647 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2881585408}
I20250902 01:59:08.375865 30922 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:07.608966 31721 raft_consensus.cc:1238] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:07.615213 31727 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:08.173892 867 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:07.592473 1191 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.581499 1130 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.674743 996 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.820178 1138 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:07.577657 1139 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.152788 31038 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:08.327025 31075 rpcz_store.cc:276] 0902 01:59:04.657040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.722154 (+2065114us) service_pool.cc:225] Handling call
0902 01:59:06.722401 (+ 247us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.320303 (+1597902us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.320370 (+ 67us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.320475 (+ 105us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.320532 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.320669 (+ 137us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.320943 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5299,"raft.queue_time_us":61339,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":3486,"thread_start_us":1584055,"threads_started":1}
I20250902 01:59:08.385609 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 394) took 4046 ms. Trace:
I20250902 01:59:07.671983 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:07.597515 31723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 901) took 1512 ms (client timeout 1910 ms). Trace:
W20250902 01:59:08.332435 30891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.340s user 0.004s sys 0.020s
I20250902 01:59:07.682518 1197 raft_consensus.cc:3053] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:08.326465 30920 rpcz_store.cc:276] 0902 01:59:06.115897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.118326 (+ 2429us) service_pool.cc:225] Handling call
0902 01:59:06.323474 (+205148us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.331008 (+ 7534us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:06.732379 (+401371us) log.cc:844] Serialized 14 byte log entry
0902 01:59:07.302193 (+569814us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.302258 (+ 65us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:07.302312 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.302392 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.302527 (+ 135us) inbound_call.cc:173] Queueing success response
0902 01:59:08.319948 (+1017421us) spinlock_profiling.cc:243] Waited 8.56 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
Metrics: {"prepare.queue_time_us":439279,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":342988,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"spinlock_wait_cycles":19677952,"thread_start_us":889955,"threads_started":3,"wal-append.queue_time_us":595313}
I20250902 01:59:07.601804 1051 raft_consensus.cc:3053] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:08.343014 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.163149 522 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:08.376892 30964 leader_election.cc:336] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.889s (SENT)
I20250902 01:59:07.590794 30997 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:08.475374 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 551) took 1304 ms. Trace:
I20250902 01:59:08.421231 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1936) took 4194 ms. Trace:
I20250902 01:59:07.666982 1206 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:08.478699 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1691) took 1728 ms. Trace:
I20250902 01:59:08.479334 31073 rpcz_store.cc:276] 0902 01:59:07.170583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.591932 (+421349us) service_pool.cc:225] Handling call
0902 01:59:08.468415 (+876483us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:07.864073 1182 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.012s sys 0.023s
W20250902 01:59:08.374699 30998 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 1954) took 2179 ms (client timeout 1774 ms). Trace:
W20250902 01:59:08.457353 30926 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 490) took 2277 ms (client timeout 2353 ms). Trace:
W20250902 01:59:08.374711 1067 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.503s user 0.013s sys 0.031s
I20250902 01:59:08.475802 31070 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
W20250902 01:59:08.482322 30996 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.494s user 0.007s sys 0.025s
W20250902 01:59:08.377084 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.447s user 0.014s sys 0.017s
W20250902 01:59:08.331277 30917 rpcz_store.cc:269] 0902 01:59:01.365148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:02.971922 (+1606774us) service_pool.cc:225] Handling call
0902 01:59:08.099933 (+5128011us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54002409d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:08.100125 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3681373440}
I20250902 01:59:08.384125 992 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:07.592175 31728 rpcz_store.cc:276] 0902 01:59:04.754870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.160773 (+2405903us) service_pool.cc:225] Handling call
0902 01:59:07.161074 (+ 301us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.591069 (+429995us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.591595 (+ 526us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54006f0754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:07.591731 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":987759488}
I20250902 01:59:08.479499 31000 rpcz_store.cc:276] 0902 01:59:04.212770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.325148 (+4112378us) service_pool.cc:225] Handling call
0902 01:59:08.325924 (+ 776us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.353962 (+ 28038us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.354002 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.358079 (+ 4077us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:08.407239 (+ 49160us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.407325 (+ 86us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.407457 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4809,"raft.queue_time_us":20959,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":5692}
I20250902 01:59:08.472626 30994 rpcz_store.cc:276] 0902 01:59:04.338748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.357183 (+4018435us) service_pool.cc:225] Handling call
0902 01:59:08.384459 (+ 27276us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.385082 (+ 623us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.385111 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.385178 (+ 67us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.385228 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.385288 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.385416 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":193}
W20250902 01:59:08.482667 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 105932us
I20250902 01:59:08.480935 30922 rpcz_store.cc:276] 0902 01:59:06.750714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.109227 (+358513us) service_pool.cc:225] Handling call
0902 01:59:08.477309 (+1368082us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:08.380735 31071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 552) took 1209 ms (client timeout 1591 ms). Trace:
I20250902 01:59:08.425972 876 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:07.612427 1167 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:08.488659 30926 rpcz_store.cc:269] 0902 01:59:06.179830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.347941 (+168111us) service_pool.cc:225] Handling call
0902 01:59:08.097153 (+1749212us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:08.457248 (+360095us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92826496}
W20250902 01:59:08.497231 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 122422us
W20250902 01:59:08.499419 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:08.501963 31067 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:08.497081 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1746) took 4470 ms. Trace:
I20250902 01:59:08.173928 618 consensus_queue.cc:1035] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.317s
I20250902 01:59:08.506515 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:08.458046 30995 rpcz_store.cc:269] 0902 01:59:04.335919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.076972 (+1741053us) service_pool.cc:225] Handling call
0902 01:59:07.586978 (+1510006us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b540004f1d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.587072 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":659330432}
W20250902 01:59:08.488332 30998 rpcz_store.cc:269] 0902 01:59:06.178277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.701925 (+523648us) service_pool.cc:225] Handling call
0902 01:59:08.353068 (+1651143us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:08.481448 31725 rpcz_store.cc:269] 0902 01:59:06.154493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.332499 (+178006us) service_pool.cc:225] Handling call
0902 01:59:07.586114 (+1253615us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b54005ee2d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.586220 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49369856}
I20250902 01:59:08.465730 30992 rpcz_store.cc:276] 0902 01:59:05.996611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.733327 (+736716us) service_pool.cc:225] Handling call
0902 01:59:07.585425 (+852098us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b54002ad5d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.585512 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139326720}
I20250902 01:59:08.506291 1148 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:08.507676 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 355289us
I20250902 01:59:08.492363 1206 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:08.507018 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 553) took 1306 ms. Trace:
I20250902 01:59:08.489131 1197 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:08.466442 1185 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:08.356536 1052 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:08.482559 31066 rpcz_store.cc:269] 0902 01:59:03.997710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.480103 (+482393us) service_pool.cc:225] Handling call
0902 01:59:07.584714 (+3104611us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54000f46d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.584802 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5109981440}
I20250902 01:59:08.357244 997 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:08.506587 991 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:08.514760 991 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:08.515157 1139 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.328123 31068 rpcz_store.cc:276] 0902 01:59:03.955416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.991602 (+ 36186us) service_pool.cc:225] Handling call
0902 01:59:06.588149 (+2596547us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff1915ce 000000000035a3c9 00007fc607018300 00007fc6070deec4 00007fc6070def31 00007fc6070a1f50 00007fc60708d199 00007fc60708cc42 00007fc600719b44
0902 01:59:06.679690 (+ 91541us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:07.714020 (+1034330us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:07.714062 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.714117 (+ 55us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:07.714158 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.098526 (+384368us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540024b3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:08.098624 (+ 98us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.098928 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":430,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":20726,"spinlock_wait_cycles":2381754368}
I20250902 01:59:08.467057 1026 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:08.511271 31070 rpcz_store.cc:276] 0902 01:59:07.200746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.579680 (+378934us) service_pool.cc:225] Handling call
0902 01:59:08.503626 (+923946us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.512050 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:08.521592 1185 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:08.512463 31067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 537) took 1691 ms (client timeout 1875 ms). Trace:
I20250902 01:59:08.508435 876 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.522845 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.520606 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:08.523957 31067 rpcz_store.cc:269] 0902 01:59:06.821477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.821854 (+ 377us) service_pool.cc:225] Handling call
0902 01:59:07.671785 (+849931us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:08.508867 (+837082us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5613952}
I20250902 01:59:08.532002 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:08.532773 31066 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.466387 1054 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:07.673153 1174 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.381820 1114 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.521874 991 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.364959 30921 rpcz_store.cc:276] 0902 01:59:06.106876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.106991 (+ 115us) service_pool.cc:225] Handling call
0902 01:59:06.108215 (+ 1224us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.108335 (+ 120us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:06.804351 (+696016us) log.cc:844] Serialized 14 byte log entry
0902 01:59:07.305512 (+501161us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.305573 (+ 61us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:07.305634 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.305710 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.305853 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":658102,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":3703,"raft.queue_time_us":387126,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":1186183,"threads_started":3,"wal-append.queue_time_us":253411,"wal-append.run_cpu_time_us":1287,"wal-append.run_wall_time_us":1284917}
I20250902 01:59:08.521064 1206 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.551378 31073 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:07.592976 1077 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.545554 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.554258 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:08.496385 31071 rpcz_store.cc:269] 0902 01:59:07.170814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.570733 (+399919us) service_pool.cc:225] Handling call
0902 01:59:08.380517 (+809784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.522554 997 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:08.555063 618 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:07.604470 993 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.523597 1204 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:08.505097 31069 rpcz_store.cc:276] 0902 01:59:03.994253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.696857 (+2702604us) service_pool.cc:225] Handling call
0902 01:59:06.716747 (+ 19890us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:06.886825 (+170078us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:06.886867 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:06.978073 (+ 91206us) log.cc:844] Serialized 26 byte log entry
0902 01:59:07.569494 (+591421us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:07.569530 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.569594 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:08.323207 (+753613us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:08.327045 (+ 3838us) raft_consensus.cc:1683] finished
0902 01:59:08.327064 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.464280 (+137216us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44211,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":15606,"spinlock_wait_cycles":24118656,"thread_start_us":659726,"threads_started":2,"wal-append.queue_time_us":515704}
I20250902 01:59:08.557819 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.557768 1197 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.554144 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.442410 31001 rpcz_store.cc:276] 0902 01:59:01.367468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.132651 (+1765183us) service_pool.cc:225] Handling call
0902 01:59:03.132991 (+ 340us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:04.330193 (+1197202us) raft_consensus.cc:1532] Early marking committed up to index 2
0902 01:59:04.551543 (+221350us) log.cc:844] Serialized 14 byte log entry
0902 01:59:04.869734 (+318191us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:04.869774 (+ 40us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:04.869802 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.582012 (+2712210us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000d7ad4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:07.582058 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.582197 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":303013,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":77,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":132743,"spinlock_wait_cycles":2752976768,"thread_start_us":509339,"threads_started":2,"wal-append.queue_time_us":1456175,"wal-append.run_cpu_time_us":9780,"wal-append.run_wall_time_us":1719907}
I20250902 01:59:08.554927 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.555480 867 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.559966 31693 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:08.516294 1138 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.574801 31073 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
W20250902 01:59:08.554217 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 493291us
I20250902 01:59:08.576373 1162 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:08.576853 1162 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:08.577255 1162 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:08.559962 789 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:08.356900 30993 rpcz_store.cc:276] 0902 01:59:03.810366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:03.810455 (+ 89us) service_pool.cc:225] Handling call
0902 01:59:03.810724 (+ 269us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:04.354893 (+544169us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:04.354928 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:04.355551 (+ 623us) log.cc:844] Serialized 52 byte log entry
0902 01:59:04.564224 (+208673us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:04.564276 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.389881 (+2825605us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54002bd754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:07.389921 (+ 40us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:07.389960 (+ 39us) raft_consensus.cc:1683] finished
0902 01:59:07.389971 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.390157 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11097,"raft.queue_time_us":61110,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":11564,"spinlock_wait_cycles":1072543104,"thread_start_us":78870,"threads_started":1,"wal-append.queue_time_us":242554,"wal-append.run_cpu_time_us":19261,"wal-append.run_wall_time_us":2173410}
I20250902 01:59:08.569114 1084 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.570511 1054 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.442575 30999 rpcz_store.cc:276] 0902 01:59:04.211264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.803844 (+2592580us) service_pool.cc:225] Handling call
0902 01:59:06.804155 (+ 311us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:06.804335 (+ 180us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:06.804510 (+ 175us) log.cc:844] Serialized 14 byte log entry
0902 01:59:07.306138 (+501628us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:07.306196 (+ 58us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:07.306252 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:07.306326 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:07.306458 (+ 132us) inbound_call.cc:173] Queueing success response
0902 01:59:07.681283 (+374825us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
Metrics: {"prepare.queue_time_us":701,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":50986,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":4778,"spinlock_wait_cycles":57272192,"thread_start_us":501179,"threads_started":2,"wal-append.queue_time_us":260434}
I20250902 01:59:08.561290 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.571231 993 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.562354 1191 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:08.567348 1051 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:08.568411 1130 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:08.587994 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 75050.1us
W20250902 01:59:08.580830 1182 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.333s user 0.013s sys 0.057s
I20250902 01:59:08.597550 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.580940 789 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:08.602813 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:08.596748 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:08.604012 789 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:08.604364 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.596729 31000 raft_consensus.cc:2374] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4. Re-sending same reply.
I20250902 01:59:08.604664 1051 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.606869 1174 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.607569 1051 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:08.608384 1197 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.602807 31075 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:08.611966 30995 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:08.612232 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.613198 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.613426 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.614065 31075 raft_consensus.cc:2466] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:08.614336 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:08.614818 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.615660 31730 raft_consensus.cc:604] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Received request to transfer leadership to TS 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:08.615892 30892 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:08.571801 522 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.616995 1204 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.618485 522 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.600858 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1931) took 4384 ms. Trace:
I20250902 01:59:08.619652 30992 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:08.619746 30996 rpcz_store.cc:276] 0902 01:59:04.211514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.819868 (+2608354us) service_pool.cc:225] Handling call
0902 01:59:06.824441 (+ 4573us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.560933 (+1736492us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.567591 (+ 6658us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.571975 (+ 4384us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.572045 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.582844 (+ 10799us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.595849 (+ 13005us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":253554,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":50274,"thread_start_us":1668667,"threads_started":1}
I20250902 01:59:08.620909 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.622154 30992 raft_consensus.cc:2408] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20250902 01:59:08.622581 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 68918.5us
I20250902 01:59:08.623137 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1841) took 4257 ms. Trace:
I20250902 01:59:08.616155 31730 raft_consensus.cc:612] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:08.623855 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:08.624104 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.79671e+06us
I20250902 01:59:08.625741 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:08.626619 992 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:08.604912 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.627379 992 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.627679 30998 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:08.623611 30999 rpcz_store.cc:276] 0902 01:59:04.365762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.621956 (+4256194us) service_pool.cc:225] Handling call
0902 01:59:08.622420 (+ 464us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.622731 (+ 311us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.622762 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.622823 (+ 61us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:08.622877 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.622941 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.623060 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.628540 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.629165 30998 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:08.629408 31071 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:08.629541 992 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:08.629766 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1845) took 4154 ms. Trace:
I20250902 01:59:08.630113 30998 rpcz_store.cc:276] 0902 01:59:04.475408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.628540 (+4153132us) service_pool.cc:225] Handling call
0902 01:59:08.628902 (+ 362us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.629555 (+ 653us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.629714 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.630125 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:08.631626 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:08.632778 1054 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250902 01:59:08.632894 30967 connection.cc:232] RPC call timeout handler was delayed by -2.0485559659999808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2359s for a response
I20250902 01:59:08.633584 1054 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250902 01:59:08.633808 30967 connection.cc:232] RPC call timeout handler was delayed by -1.6648225970000112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2343s for a response
W20250902 01:59:08.634305 30967 connection.cc:232] RPC call timeout handler was delayed by -1.4176063520000071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2739s for a response
W20250902 01:59:08.634694 30967 connection.cc:232] RPC call timeout handler was delayed by -1.3321559659999593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3155s for a response
W20250902 01:59:08.635041 30967 connection.cc:232] RPC call timeout handler was delayed by -1.2619063519999827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2912s for a response
I20250902 01:59:08.632489 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.640162 31071 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
W20250902 01:59:08.636018 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:08.636969 993 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:08.643828 992 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:08.659026 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.627238 30889 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:08.652038 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1948) took 3864 ms. Trace:
I20250902 01:59:08.636584 1141 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.644429 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1935) took 4348 ms. Trace:
I20250902 01:59:08.644336 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 888) took 3779 ms. Trace:
I20250902 01:59:08.661072 30998 rpcz_store.cc:276] 0902 01:59:04.787952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.630880 (+3842928us) service_pool.cc:225] Handling call
0902 01:59:08.631235 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.631494 (+ 259us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.631523 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.651701 (+ 20178us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:08.651778 (+ 77us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.651852 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.651981 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.663022 31066 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:08.655184 31068 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
W20250902 01:59:08.668546 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 69603.7us
I20250902 01:59:08.661345 31727 rpcz_store.cc:276] 0902 01:59:04.765958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.614487 (+2848529us) service_pool.cc:225] Handling call
0902 01:59:07.614842 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.489724 (+874882us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.545030 (+ 55306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.669523 31072 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:08.671085 31070 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:08.661283 30997 rpcz_store.cc:276] 0902 01:59:04.212551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.590411 (+3377860us) service_pool.cc:225] Handling call
0902 01:59:07.590640 (+ 229us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.510120 (+919480us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.561131 (+ 51011us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:08.679818 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 52776us
I20250902 01:59:08.680739 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:08.562366 31723 rpcz_store.cc:269] 0902 01:59:06.085151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.709721 (+624570us) service_pool.cc:225] Handling call
0902 01:59:07.597312 (+887591us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b54005901d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:07.597458 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112528640}
I20250902 01:59:08.653457 789 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.682458 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.683326 31723 raft_consensus.cc:2391] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.644593 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.643769 1077 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.659319 992 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:08.653419 993 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:08.689502 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.687950 30992 pending_rounds.cc:85] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 5
W20250902 01:59:08.691613 30964 leader_election.cc:336] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.883s (SENT)
I20250902 01:59:08.693581 30992 pending_rounds.cc:107] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250902 01:59:08.696193 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 891) took 3924 ms. Trace:
I20250902 01:59:08.694864 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:08.696559 31729 rpcz_store.cc:276] 0902 01:59:04.768711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.542693 (+3773982us) service_pool.cc:225] Handling call
0902 01:59:08.602737 (+ 60044us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.686465 (+ 83728us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.686489 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.686559 (+ 70us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:08.693233 (+ 6674us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.693304 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.693411 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.698770 30993 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:08.689338 992 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:08.654896 1052 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.691943 993 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.668013 618 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.702021 31727 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.696449 30992 raft_consensus.cc:2887] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:08.684993 832 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:08.703456 30919 raft_consensus.cc:1238] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:59:08.703811 1212 consensus_queue.cc:1046] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250902 01:59:08.704380 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.712078 31066 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:08.713449 31075 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:08.715895 1084 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.716976 1114 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.717593 996 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:08.717998 996 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:08.718227 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1952) took 2551 ms. Trace:
I20250902 01:59:08.718328 996 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.723644 1027 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:08.724048 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1953) took 2553 ms. Trace:
I20250902 01:59:08.724300 1024 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.724864 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 550) took 1554 ms. Trace:
I20250902 01:59:08.724071 1027 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:08.725320 31070 rpcz_store.cc:276] 0902 01:59:07.170367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.684368 (+1514001us) service_pool.cc:225] Handling call
0902 01:59:08.699693 (+ 15325us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.724548 (+ 24855us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.724576 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.724616 (+ 40us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:08.724640 (+ 24us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.724690 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.724801 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.726228 1024 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.726846 31070 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:08.728217 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 555) took 1070 ms. Trace:
I20250902 01:59:08.727619 30998 rpcz_store.cc:276] 0902 01:59:06.166954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.670198 (+2503244us) service_pool.cc:225] Handling call
0902 01:59:08.697924 (+ 27726us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.717897 (+ 19973us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.717921 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.717983 (+ 62us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:08.718010 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.718065 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.718160 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.728595 31067 rpcz_store.cc:276] 0902 01:59:07.657470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.692175 (+1034705us) service_pool.cc:225] Handling call
0902 01:59:08.727638 (+ 35463us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.727904 (+ 266us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.727930 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.727976 (+ 46us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.728009 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.728066 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.728166 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.729516 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:08.730181 31067 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250902 01:59:08.733225 30993 pending_rounds.cc:85] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 6
I20250902 01:59:08.733489 30993 pending_rounds.cc:107] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250902 01:59:08.733695 30993 raft_consensus.cc:2887] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:08.724519 30997 rpcz_store.cc:276] 0902 01:59:06.170185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.687678 (+2517493us) service_pool.cc:225] Handling call
0902 01:59:08.709084 (+ 21406us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.723791 (+ 14707us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.723815 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.723850 (+ 35us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:08.723877 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.723922 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.724005 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.736241 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:08.737231 30997 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.739614 30925 raft_consensus.cc:2391] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:08.741405 867 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.742151 832 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.743614 1141 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.724114 1060 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:08.711499 1185 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:08.746096 31074 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 545) took 1595 ms (client timeout 1728 ms). Trace:
I20250902 01:59:08.746382 1185 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:08.746469 31074 rpcz_store.cc:269] 0902 01:59:07.150288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.150895 (+ 607us) service_pool.cc:225] Handling call
0902 01:59:08.745835 (+1594940us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b5400257e54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:08.745955 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12879232}
I20250902 01:59:08.748987 31001 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:08.751431 1077 raft_consensus.cc:3053] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:08.751940 1077 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:08.727451 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 559) took 1028 ms. Trace:
I20250902 01:59:08.767693 31070 rpcz_store.cc:276] 0902 01:59:07.699416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.726262 (+1026846us) service_pool.cc:225] Handling call
0902 01:59:08.726605 (+ 343us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.727196 (+ 591us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.727385 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.774199 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 548) took 1542 ms. Trace:
I20250902 01:59:08.776243 31066 rpcz_store.cc:276] 0902 01:59:07.169921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.680229 (+1510308us) service_pool.cc:225] Handling call
0902 01:59:08.700205 (+ 19976us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.712540 (+ 12335us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.712783 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:08.730182 30998 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:08.783105 30967 leader_election.cc:336] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.189s (SENT)
I20250902 01:59:08.752971 1077 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.800675 1077 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:08.801597 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 606) took 4039 ms. Trace:
I20250902 01:59:08.802079 31721 rpcz_store.cc:276] 0902 01:59:04.763141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.571411 (+2808270us) service_pool.cc:225] Handling call
0902 01:59:07.608799 (+ 37388us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.504315 (+895516us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.801278 (+296963us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b54005901d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:08.801425 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85259136}
I20250902 01:59:08.804831 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:08.710289 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.805241 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 493) took 2548 ms. Trace:
I20250902 01:59:08.806138 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1944) took 4126 ms. Trace:
I20250902 01:59:08.806486 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.803413 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.807250 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:49554 (request call id 5) took 2120 ms. Trace:
I20250902 01:59:08.807789 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.808017 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.808650 31721 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:08.808961 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.809428 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.805922 30919 rpcz_store.cc:276] 0902 01:59:06.257336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.481836 (+224500us) service_pool.cc:225] Handling call
0902 01:59:06.482108 (+ 272us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.710603 (+2228495us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.804950 (+ 94347us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b540018f954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:08.805175 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4830032256}
I20250902 01:59:08.710947 789 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.806640 30999 rpcz_store.cc:276] 0902 01:59:04.680009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.628710 (+3948701us) service_pool.cc:225] Handling call
0902 01:59:08.629120 (+ 410us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.634524 (+ 5404us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.634567 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.635064 (+ 497us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.692583 (+ 57519us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:08.692635 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.692700 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:08.805922 (+113222us) raft_consensus.cc:1683] finished
0902 01:59:08.805943 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.806067 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4067,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":130,"thread_start_us":57099,"threads_started":1,"wal-append.queue_time_us":168431}
I20250902 01:59:08.810078 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.796829 996 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:08.812659 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 181224us
W20250902 01:59:08.811097 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 61851.9us
W20250902 01:59:08.812529 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 131637us
I20250902 01:59:08.815737 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:08.813716 30964 leader_election.cc:336] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.039s (SENT)
I20250902 01:59:08.816306 31069 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:08.807631 31730 rpcz_store.cc:276] 0902 01:59:06.686378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.827263 (+140885us) service_pool.cc:225] Handling call
0902 01:59:08.807021 (+1979758us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54004d53d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212a813 00007fc6070901ba 00007fc60072a1c4 00007fc60072a152 00007fc60072a081 00007fc600729ffc 00007fc6007292b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:08.807137 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3949518720}
I20250902 01:59:08.821910 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:08.822466 31073 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:08.816133 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 608) took 4040 ms. Trace:
I20250902 01:59:08.808991 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.816506 31721 raft_consensus.cc:2391] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:08.826797 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 133080us
I20250902 01:59:08.827152 832 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.826031 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 556) took 1159 ms. Trace:
W20250902 01:59:08.830060 1219 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20250902 01:59:08.831734 30923 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:08.832307 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.837459 31072 rpcz_store.cc:276] 0902 01:59:07.663213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.556510 (+893297us) service_pool.cc:225] Handling call
0902 01:59:08.822079 (+265569us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b540003b1d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:08.822222 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110441216}
I20250902 01:59:08.783248 1060 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.823355 31725 rpcz_store.cc:276] 0902 01:59:04.771940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.535691 (+3763751us) service_pool.cc:225] Handling call
0902 01:59:08.536227 (+ 536us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.536964 (+ 737us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.536997 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.537638 (+ 641us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.672105 (+134467us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:08.672404 (+ 299us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.672491 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:08.812401 (+139910us) raft_consensus.cc:1683] finished
0902 01:59:08.812423 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.812708 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2880,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":271,"thread_start_us":133881,"threads_started":1,"wal-append.queue_time_us":246072}
I20250902 01:59:08.843154 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:08.839633 31729 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 2.
I20250902 01:59:08.885460 1051 consensus_queue.cc:1046] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:08.886590 1051 raft_consensus.cc:3053] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:59:08.886884 1051 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:08.876772 867 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:08.888105 867 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.888499 1052 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.879839 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.889343 31725 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.889536 1162 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.890110 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.889966 1148 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.891134 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.892076 31722 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.892107 1051 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.893344 1162 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.898618 876 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:08.899740 618 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:08.917953 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:08.900543 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:08.902208 996 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.902668 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 558) took 1205 ms. Trace:
I20250902 01:59:08.903927 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.904959 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:08.905651 867 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:08.907936 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.908572 1148 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:08.908936 993 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:08.911453 31038 connection.cc:232] RPC call timeout handler was delayed by -1.9320420149999791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3053s for a response
W20250902 01:59:08.933499 31038 connection.cc:232] RPC call timeout handler was delayed by -1.9320420149999791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3053s for a response
W20250902 01:59:08.933836 31038 connection.cc:232] RPC call timeout handler was delayed by -1.1207970939999541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20250902 01:59:08.934001 31038 connection.cc:232] RPC call timeout handler was delayed by -1.095597094000027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20250902 01:59:08.934146 31038 connection.cc:232] RPC call timeout handler was delayed by -1.0262970940000287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
W20250902 01:59:08.934316 31038 connection.cc:232] RPC call timeout handler was delayed by -1.009197094000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17250000000000001s for a response
I20250902 01:59:08.914551 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.917162 31727 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:08.899677 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:08.919899 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 50601.2us
I20250902 01:59:08.920149 30919 raft_consensus.cc:1238] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
W20250902 01:59:08.930708 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 60467.9us
I20250902 01:59:08.931331 31730 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.932349 30995 raft_consensus.cc:2391] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:08.928260 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:08.932989 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1689) took 2177 ms. Trace:
I20250902 01:59:08.938828 31725 raft_consensus.cc:2408] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:08.899423 1130 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.941462 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 395) took 4600 ms. Trace:
I20250902 01:59:08.938493 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.941928 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.943008 31695 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:08.943818 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:08.914454 1191 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.945420 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.946408 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.946652 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:08.942302 30996 rpcz_store.cc:276] 0902 01:59:04.338973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.621444 (+4282471us) service_pool.cc:225] Handling call
0902 01:59:08.621869 (+ 425us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.622168 (+ 299us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:08.816675 (+194507us) log.cc:844] Serialized 14 byte log entry
0902 01:59:08.939036 (+122361us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.939081 (+ 45us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.939117 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.939171 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.939263 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23735,"prepare.queue_time_us":203011,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":1513,"thread_start_us":198421,"threads_started":2}
I20250902 01:59:08.992110 31727 raft_consensus.cc:2391] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:08.942153 1052 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:08.942341 1162 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:08.941738 618 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:09.004065 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.005617 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:09.011054 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 66205.8us
I20250902 01:59:08.994467 1027 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:08.922130 31071 rpcz_store.cc:276] 0902 01:59:07.697353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.705197 (+1007844us) service_pool.cc:225] Handling call
0902 01:59:08.727056 (+ 21859us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:08.732501 (+ 5445us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.732533 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:08.732598 (+ 65us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.732637 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.869389 (+136752us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b5400259c54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:08.873580 (+ 4191us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.879752 (+ 6172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12039552}
I20250902 01:59:08.951912 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:08.943671 1206 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:09.003105 1224 log.cc:927] Time spent T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.204s user 0.000s sys 0.003s
I20250902 01:59:09.008605 30892 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:09.003655 903 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:09.020861 31728 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:09.004050 993 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:09.004587 1228 log.cc:927] Time spent T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
W20250902 01:59:09.005088 1223 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20250902 01:59:09.027645 991 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:09.028086 991 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:09.028264 31722 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 600) took 4279 ms (client timeout 1624 ms). Trace:
I20250902 01:59:09.028533 991 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:09.028658 31722 rpcz_store.cc:269] 0902 01:59:04.749068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:04.762439 (+ 13371us) service_pool.cc:225] Handling call
0902 01:59:07.328148 (+2565709us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:09.028033 (+1699885us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54004b05d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.028161 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3023589504}
I20250902 01:59:09.028100 1191 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:09.005239 31000 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250902 01:59:09.030824 1027 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.031868 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1688) took 2283 ms. Trace:
I20250902 01:59:09.032272 30917 rpcz_store.cc:276] 0902 01:59:06.748913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.552694 (+1803781us) service_pool.cc:225] Handling call
0902 01:59:08.687591 (+134897us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.703801 (+ 16210us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.703837 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.704337 (+ 500us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.917385 (+213048us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:08.917421 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.917506 (+ 85us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.031603 (+114097us) raft_consensus.cc:1683] finished
0902 01:59:09.031627 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.031791 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58380,"raft.queue_time_us":2841,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":3261,"thread_start_us":212687,"threads_started":1,"wal-append.queue_time_us":224525}
I20250902 01:59:09.033522 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1949) took 2907 ms. Trace:
I20250902 01:59:09.033845 31000 rpcz_store.cc:276] 0902 01:59:06.126133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.669375 (+2543242us) service_pool.cc:225] Handling call
0902 01:59:09.004925 (+335550us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:09.030003 (+ 25078us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.033369 (+ 3366us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b5400177d50. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.033473 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":719148800}
I20250902 01:59:09.035099 31038 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.006162 30924 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:09.035854 789 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:09.008577 31724 raft_consensus.cc:1238] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250902 01:59:08.993959 31729 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:08.995170 1113 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:08.937948 30922 rpcz_store.cc:276] 0902 01:59:06.749182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.754075 (+2004893us) service_pool.cc:225] Handling call
0902 01:59:08.754428 (+ 353us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.754856 (+ 428us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.754895 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.755381 (+ 486us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.839703 (+ 84322us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:08.839754 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.839974 (+ 220us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:08.926638 (+ 86664us) raft_consensus.cc:1683] finished
0902 01:59:08.926667 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:08.926817 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12348,"raft.queue_time_us":3132,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"thread_start_us":83636,"threads_started":1,"wal-append.queue_time_us":162349}
W20250902 01:59:08.994783 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 51895.2us
I20250902 01:59:09.037694 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 607) took 4271 ms. Trace:
I20250902 01:59:09.037626 1113 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.038087 31695 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.038178 31724 rpcz_store.cc:276] 0902 01:59:04.766275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.376827 (+3610552us) service_pool.cc:225] Handling call
0902 01:59:08.377565 (+ 738us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:09.036713 (+659148us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.037443 (+ 730us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54004d8354. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:09.037631 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1450499328}
I20250902 01:59:09.040134 1113 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.010638 31725 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:09.040776 1113 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:09.041107 1113 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:09.018006 31723 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:09.018992 1049 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.041812 1026 consensus_queue.cc:1046] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:09.041977 997 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.042624 1026 raft_consensus.cc:3053] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:09.042963 30921 raft_consensus.cc:1238] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.044428 1130 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:08.993638 996 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.052s
I20250902 01:59:09.022744 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1686) took 2286 ms. Trace:
I20250902 01:59:09.042986 1026 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:09.046353 1026 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.046921 876 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.023075 1206 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:09.023749 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1675) took 2316 ms. Trace:
I20250902 01:59:09.045758 30918 rpcz_store.cc:276] 0902 01:59:06.736346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.482388 (+746042us) service_pool.cc:225] Handling call
0902 01:59:07.558192 (+ 75804us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.562839 (+1004647us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.568917 (+ 6078us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.615832 (+ 46915us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.690963 (+ 75131us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:08.691001 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.022244 (+331243us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54002a21d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:09.022298 (+ 54us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.022351 (+ 53us) raft_consensus.cc:1683] finished
0902 01:59:09.022365 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.022567 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61360,"raft.queue_time_us":61911,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":178,"spinlock_wait_cycles":1734773632,"thread_start_us":243807,"threads_started":2,"wal-append.queue_time_us":90442}
I20250902 01:59:09.024533 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:09.025367 30923 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1685) took 2289 ms (client timeout 2606 ms). Trace:
I20250902 01:59:09.036266 789 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:09.048027 30925 rpcz_store.cc:276] 0902 01:59:06.711909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.167515 (+455606us) service_pool.cc:225] Handling call
0902 01:59:09.023504 (+1855989us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540021dc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.023696 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756235008}
W20250902 01:59:09.049221 30923 rpcz_store.cc:269] 0902 01:59:06.735987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:07.108194 (+372207us) service_pool.cc:225] Handling call
0902 01:59:09.025193 (+1916999us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400337a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.025317 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3959017216}
I20250902 01:59:09.023640 1077 consensus_queue.cc:1046] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:09.049901 789 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.050621 996 raft_consensus.cc:3053] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
W20250902 01:59:09.082000 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.064150 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:09.091426 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:09.091964 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.065774 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.067992 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.037245 31038 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:09.057446 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.056265 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:09.063782 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.059075 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:09.092978 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.093240 30919 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.066045 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.058130 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:09.062781 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.055706 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:09.050921 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.011094 1060 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.092676 30994 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:09.057591 31726 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:09.050851 1113 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.097858 30923 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.106853 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.022154 1051 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:59:09.108274 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.054821 1026 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:09.108836 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 71704.5us
I20250902 01:59:09.054337 1138 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.091217 996 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:09.115247 522 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:09.115741 522 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:09.116166 522 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.115833 996 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.117566 996 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:09.111433 30964 leader_election.cc:336] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.207s (SENT)
I20250902 01:59:09.111474 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.110314 30925 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.112342 31000 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:09.090652 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:09.119716 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1843) took 4753 ms. Trace:
I20250902 01:59:09.059048 1162 consensus_queue.cc:1046] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20250902 01:59:09.059278 1206 consensus_queue.cc:1046] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20250902 01:59:09.051030 1077 consensus_queue.cc:1046] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20250902 01:59:09.094480 1130 consensus_queue.cc:1046] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250902 01:59:09.129976 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.120529 30992 rpcz_store.cc:276] 0902 01:59:04.366206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.622908 (+4256702us) service_pool.cc:225] Handling call
0902 01:59:08.623440 (+ 532us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:08.784020 (+160580us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.784061 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:08.784663 (+ 602us) log.cc:844] Serialized 52 byte log entry
0902 01:59:08.879690 (+ 95027us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:08.879731 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:08.879790 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.119520 (+239730us) raft_consensus.cc:1683] finished
0902 01:59:09.119537 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.119665 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1426,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":29226,"thread_start_us":94568,"threads_started":1,"wal-append.queue_time_us":152265}
I20250902 01:59:09.132319 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.051638 992 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.130214 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.133298 30994 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:09.130703 31001 raft_consensus.cc:2408] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:09.090358 1052 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.119295 997 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.130033 30997 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:09.137866 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:09.138022 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 89519.8us
I20250902 01:59:09.137524 1151 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:09.140252 1151 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.145247 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.145802 30993 raft_consensus.cc:3058] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.146572 31000 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:09.148661 1052 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.151770 31038 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.151837 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.153856 30998 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:09.153034 1138 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:09.156370 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.157120 30992 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.110595 618 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:09.158893 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 61454.9us
I20250902 01:59:09.159101 30993 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:09.128916 31721 raft_consensus.cc:2374] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4. Re-sending same reply.
I20250902 01:59:09.470211 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.302898 30992 raft_consensus.cc:2466] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:09.160219 1138 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:09.167827 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:09.470111 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 305531us
W20250902 01:59:09.262383 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 123800us
I20250902 01:59:09.411499 30994 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:09.201892 31000 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:09.479118 30923 raft_consensus.cc:1238] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:09.246460 30919 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:09.175302 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:09.497467 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 351069us
W20250902 01:59:09.329522 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 56894.4us
I20250902 01:59:09.172737 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:09.419405 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.175945 30925 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:09.262668 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:09.496891 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 343912us
I20250902 01:59:09.472458 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:09.522643 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.471040 31728 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:09.515990 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 549) took 2326 ms. Trace:
W20250902 01:59:09.172016 31726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 895) took 3095 ms (client timeout 2155 ms). Trace:
I20250902 01:59:09.512512 30918 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:09.510833 31729 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:09.512609 30922 raft_consensus.cc:1238] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.520161 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1692) took 2740 ms. Trace:
I20250902 01:59:09.264345 31721 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.164037 903 raft_consensus.cc:3053] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:59:09.252466 618 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.168148 535 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250902 01:59:09.536968 31726 rpcz_store.cc:269] 0902 01:59:06.076963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:06.755371 (+678408us) service_pool.cc:225] Handling call
0902 01:59:09.171789 (+2416418us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54005abed4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.171922 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5293281536}
I20250902 01:59:09.409379 1241 raft_consensus.cc:3053] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:09.540205 30926 rpcz_store.cc:276] 0902 01:59:06.756723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.745083 (+1988360us) service_pool.cc:225] Handling call
0902 01:59:08.745432 (+ 349us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.809252 (+ 63820us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.809283 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.809712 (+ 429us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.825505 (+ 15793us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:09.114537 (+289032us) log.cc:844] Serialized 14 byte log entry
0902 01:59:09.496759 (+382222us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.496942 (+ 183us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.497015 (+ 73us) raft_consensus.cc:1683] finished
0902 01:59:09.497027 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.497132 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5453,"prepare.queue_time_us":288982,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":25442,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":9422,"spinlock_wait_cycles":9422592,"thread_start_us":591286,"threads_started":3,"wal-append.queue_time_us":15700}
I20250902 01:59:09.172600 992 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:09.548836 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 286311us
I20250902 01:59:09.167166 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.496914 1141 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:09.536831 31068 rpcz_store.cc:276] 0902 01:59:07.170140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:08.670460 (+1500320us) service_pool.cc:225] Handling call
0902 01:59:08.696367 (+ 25907us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:08.794915 (+ 98548us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:08.794956 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:08.824503 (+ 29547us) log.cc:844] Serialized 26 byte log entry
0902 01:59:08.991837 (+167334us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:09.162402 (+170565us) log.cc:844] Serialized 14 byte log entry
0902 01:59:09.495804 (+333402us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.496289 (+ 485us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.496340 (+ 51us) raft_consensus.cc:1683] finished
0902 01:59:09.496351 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.496478 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":229287,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":8818,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":4835,"thread_start_us":473291,"threads_started":3,"wal-append.queue_time_us":158671}
I20250902 01:59:09.525731 1138 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.548550 31038 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.552623 31729 raft_consensus.cc:1238] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.264485 993 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:09.170158 522 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.254972 789 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.332880 1052 consensus_queue.cc:1046] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.216s
W20250902 01:59:09.556106 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 72694us
I20250902 01:59:09.536808 1051 consensus_queue.cc:1046] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20250902 01:59:09.557884 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.545060 1148 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20250902 01:59:09.554209 1049 consensus_queue.cc:1046] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20250902 01:59:09.564591 978 consensus_queue.cc:1046] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20250902 01:59:09.558315 31692 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.550406 832 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:09.558629 31725 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:09.481974 991 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.571146 30889 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:09.572022 31728 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.594863 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 894) took 3519 ms. Trace:
I20250902 01:59:09.595181 31728 rpcz_store.cc:276] 0902 01:59:06.075401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.536192 (+3460791us) service_pool.cc:225] Handling call
0902 01:59:09.551502 (+ 15310us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.594633 (+ 43131us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.594813 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:09.573570 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.553553 1241 raft_consensus.cc:738] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:09.546630 903 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:09.577831 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.577817 31730 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:59:09.598443 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 461) took 1039 ms. Trace:
I20250902 01:59:09.598703 31730 rpcz_store.cc:276] 0902 01:59:08.558792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.547324 (+988532us) service_pool.cc:225] Handling call
0902 01:59:09.557537 (+ 10213us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.598194 (+ 40657us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.598396 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:09.579073 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 460) took 1017 ms. Trace:
W20250902 01:59:09.580063 1243 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20250902 01:59:09.581944 1244 raft_consensus.cc:2802] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:09.582656 535 tablet_bootstrap.cc:492] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Bootstrap complete.
I20250902 01:59:09.593375 1051 raft_consensus.cc:3053] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
W20250902 01:59:09.583206 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.558259 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 609) took 4777 ms. Trace:
I20250902 01:59:09.584980 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 458) took 1027 ms. Trace:
I20250902 01:59:09.586472 832 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:09.587172 618 raft_consensus.cc:3053] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:09.556792 1191 consensus_queue.cc:1046] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250902 01:59:09.589926 31038 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.575881 993 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:09.597177 31726 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:09.599278 31729 rpcz_store.cc:276] 0902 01:59:08.558477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.545342 (+986865us) service_pool.cc:225] Handling call
0902 01:59:09.551155 (+ 5813us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.575983 (+ 24828us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.576151 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:09.576777 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.599726 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 610) took 4811 ms. Trace:
I20250902 01:59:09.600069 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.600497 30921 raft_consensus.cc:1238] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:09.557540 1247 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:09.603991 31727 rpcz_store.cc:276] 0902 01:59:08.557633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.032653 (+475020us) service_pool.cc:225] Handling call
0902 01:59:09.557749 (+525096us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:09.583041 (+ 25292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133244160}
I20250902 01:59:09.607025 522 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:09.607743 1113 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.608219 535 ts_tablet_manager.cc:1397] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Time spent bootstrapping tablet: real 1.929s user 0.048s sys 0.041s
W20250902 01:59:09.608656 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.608796 1244 raft_consensus.cc:695] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:09.603976 1051 raft_consensus.cc:738] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:09.610083 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 60730.1us
I20250902 01:59:09.610298 1174 raft_consensus.cc:3053] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:09.608489 31721 rpcz_store.cc:276] 0902 01:59:04.779863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.263650 (+4483787us) service_pool.cc:225] Handling call
0902 01:59:09.264048 (+ 398us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.548447 (+284399us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.557053 (+ 8606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:09.611629 1024 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:59:09.615252 991 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.616230 1148 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:09.616505 903 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.618804 1138 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.623235 31068 raft_consensus.cc:3058] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.621233 618 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:09.596920 31723 raft_consensus.cc:2408] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:09.627096 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 466) took 1066 ms. Trace:
I20250902 01:59:09.659204 31726 rpcz_store.cc:276] 0902 01:59:08.560338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.568138 (+1007800us) service_pool.cc:225] Handling call
0902 01:59:09.578390 (+ 10252us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.622693 (+ 44303us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.627044 (+ 4351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:09.641240 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.621475 832 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:09.652298 30926 raft_consensus.cc:1238] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 2. Current term is 3. Ops: []
I20250902 01:59:09.627180 993 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.623100 31725 rpcz_store.cc:276] 0902 01:59:04.786294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.536241 (+4749947us) service_pool.cc:225] Handling call
0902 01:59:09.558517 (+ 22276us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.594154 (+ 35637us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.595254 (+ 1100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:09.668779 31722 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.626883 1130 raft_consensus.cc:2762] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250902 01:59:09.654690 1148 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:09.661615 903 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:09.663026 1138 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:09.663745 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.664435 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.642582 1174 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:09.653307 1024 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:09.673894 30889 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.675199 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.683440 31070 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:09.684521 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.685377 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.683449 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.688467 31073 raft_consensus.cc:2408] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250902 01:59:09.683876 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.691352 31072 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.686748 1024 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:09.686309 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.700590 30926 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:09.646219 31729 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:59:09.702237 31729 raft_consensus.cc:604] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.702503 31729 raft_consensus.cc:612] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250902 01:59:09.701910 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.704783 31066 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.668008 1051 consensus_queue.cc:260] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.703999 30964 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:09.711792 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 64921.7us
I20250902 01:59:09.705744 1191 consensus_queue.cc:1046] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20250902 01:59:09.713164 1191 raft_consensus.cc:3053] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:09.713373 30889 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.721593 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.722144 31075 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:09.722996 992 consensus_queue.cc:1046] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:59:09.724172 31069 raft_consensus.cc:1238] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250902 01:59:09.674013 1247 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.724232 992 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:09.725082 992 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:09.725761 31069 raft_consensus.cc:1238] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:09.726444 1247 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:09.726024 992 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.727237 31070 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:09.727394 992 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.682753 1241 consensus_queue.cc:260] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.730612 535 raft_consensus.cc:357] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:59:09.731989 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.718631 1174 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.736671 1174 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.737064 1151 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.737241 31729 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:59:09.737797 31729 raft_consensus.cc:604] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.738126 31729 raft_consensus.cc:612] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:09.737798 618 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.735762 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.739758 618 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.740123 31067 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.743652 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 472) took 1131 ms. Trace:
I20250902 01:59:09.743971 31728 rpcz_store.cc:276] 0902 01:59:08.612737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.596001 (+983264us) service_pool.cc:225] Handling call
0902 01:59:09.602224 (+ 6223us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:09.619692 (+ 17468us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:09.619721 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:09.620151 (+ 430us) log.cc:844] Serialized 26 byte log entry
0902 01:59:09.677551 (+ 57400us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:09.677603 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.677701 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.743441 (+ 65740us) raft_consensus.cc:1683] finished
0902 01:59:09.743459 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.743602 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17131,"raft.queue_time_us":17303,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":66,"thread_start_us":57018,"threads_started":1,"wal-append.queue_time_us":91689}
I20250902 01:59:09.717578 1148 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.713464 1191 raft_consensus.cc:738] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:09.745785 1148 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.746426 1191 consensus_queue.cc:260] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.747675 1191 raft_consensus.cc:3058] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:09.749658 1049 consensus_queue.cc:1162] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4372b4c4c866431faa1bf3e91fd47a88" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250902 01:59:09.751920 1138 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:09.752465 1138 raft_consensus.cc:695] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:09.752573 31728 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:49554
I20250902 01:59:09.753149 31728 raft_consensus.cc:604] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.753538 1138 consensus_queue.cc:237] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:09.755959 1113 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.757551 993 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.762224 31072 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:09.705837 991 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:09.717373 997 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:09.764353 30889 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:09.766024 997 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:09.767103 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 52281.5us
I20250902 01:59:09.753549 31728 raft_consensus.cc:612] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:09.770951 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.771512 31072 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:09.701740 789 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:09.783579 30892 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.709506 31071 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:09.785666 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 465) took 1225 ms. Trace:
I20250902 01:59:09.785998 31722 rpcz_store.cc:276] 0902 01:59:08.559981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.548520 (+988539us) service_pool.cc:225] Handling call
0902 01:59:09.557718 (+ 9198us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.669144 (+111426us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.785468 (+116324us) spinlock_profiling.cc:243] Waited 99.8 ms on lock 0x7b540061f154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:09.785618 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":229458816}
I20250902 01:59:09.780690 997 consensus_queue.cc:1046] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20250902 01:59:09.759379 993 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:09.793181 1147 raft_consensus.cc:3053] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:09.797472 867 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:09.816959 867 raft_consensus.cc:695] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:09.817929 867 consensus_queue.cc:237] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.799129 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250902 01:59:09.806572 30892 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.821986 1141 consensus_queue.cc:1046] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:59:09.731449 535 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Initialized, Role: LEARNER
I20250902 01:59:09.743204 832 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.813045 1147 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:09.825460 1185 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:09.831557 31067 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:09.831626 31068 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:09.836933 31075 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:09.837076 31066 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:09.833809 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250902 01:59:09.842489 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 252651us
W20250902 01:59:09.843197 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 142170us
I20250902 01:59:09.845554 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.846051 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.852885 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.899315 31069 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.844897 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.903530 31068 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:09.845173 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.844255 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.892865 1185 raft_consensus.cc:695] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:09.853188 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.895910 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.827960 992 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:09.896726 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 68233.4us
I20250902 01:59:09.909026 31073 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.703544 1244 consensus_queue.cc:237] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:09.915804 30872 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 558db87de6fa4361bf0ca91e27d34943, TS 975ecd397f5146409baac87e56a302e9 -> TS 01f6731ce5044a6a948a5ae04da80e6f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:09.916132 30872 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 558db87de6fa4361bf0ca91e27d34943, TS 975ecd397f5146409baac87e56a302e9 -> TS 01f6731ce5044a6a948a5ae04da80e6f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:09.919036 31071 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.919970 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.920518 31071 raft_consensus.cc:2374] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4. Re-sending same reply.
I20250902 01:59:09.921291 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.844100 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.899024 30964 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:09.901273 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:09.901785 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:09.922469 31075 raft_consensus.cc:2391] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.922488 31722 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:09.900476 31067 raft_consensus.cc:2391] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.906019 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.923411 31068 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.905637 31066 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:09.914433 1185 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.931774 31727 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:09.915405 31000 raft_consensus.cc:2366] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:09.932669 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.933260 31067 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:09.934041 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.895278 535 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:09.935118 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.936061 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:09.936782 30965 leader_election.cc:336] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.609s (SENT)
I20250902 01:59:09.936209 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.937794 535 ts_tablet_manager.cc:1428] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Time spent starting tablet: real 0.283s user 0.009s sys 0.000s
I20250902 01:59:09.938751 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.939271 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.937822 31722 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:09.895143 1147 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:09.940192 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.940657 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.940699 1147 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.940909 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.941334 31067 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:09.941854 31069 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.942611 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.922009 31071 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.926048 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 473) took 1313 ms. Trace:
I20250902 01:59:09.928303 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.943581 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.943819 31728 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:09.944070 31069 raft_consensus.cc:2374] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4. Re-sending same reply.
I20250902 01:59:09.928949 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.944859 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.943373 31730 rpcz_store.cc:276] 0902 01:59:08.613059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.599150 (+986091us) service_pool.cc:225] Handling call
0902 01:59:09.621274 (+ 22124us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:09.635476 (+ 14202us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:09.635523 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:09.671604 (+ 36081us) log.cc:844] Serialized 26 byte log entry
0902 01:59:09.781103 (+109499us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:09.781139 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.781197 (+ 58us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:09.925795 (+144598us) raft_consensus.cc:1683] finished
0902 01:59:09.925820 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:09.925986 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":433,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":6671,"thread_start_us":80027,"threads_started":1,"wal-append.queue_time_us":189157}
I20250902 01:59:09.929073 30892 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.929642 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:09.936821 31068 raft_consensus.cc:2391] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:09.930044 1027 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:09.947261 30892 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.899859 991 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20250902 01:59:09.900300 31072 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
W20250902 01:59:09.968039 31038 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.629s (SENT)
I20250902 01:59:09.963526 30892 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.940718 31075 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:09.965098 31069 raft_consensus.cc:2391] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:09.965086 30965 leader_election.cc:336] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.638s (SENT)
I20250902 01:59:09.976150 30892 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.979434 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:09.979669 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.980299 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:09.980680 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:09.980983 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:09.980062 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:09.920197 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:09.985355 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 907) took 1149 ms. Trace:
I20250902 01:59:09.985854 31727 rpcz_store.cc:276] 0902 01:59:08.836969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.635300 (+798331us) service_pool.cc:225] Handling call
0902 01:59:09.931617 (+296317us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:09.932151 (+ 534us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.985121 (+ 52970us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54006a9fd0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:09.985290 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636094336}
I20250902 01:59:09.990857 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.909595 992 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:09.991348 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.992527 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.977324 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.993692 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:09.995491 31073 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:09.995927 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:09.994419 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 94045.3us
I20250902 01:59:09.996734 31071 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:10.001503 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.995029 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:09.981223 1141 raft_consensus.cc:2802] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:09.978205 1246 raft_consensus.cc:2802] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:10.007694 31069 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:10.011317 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.011617 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:10.015501 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.015779 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.015918 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.054121 31072 raft_consensus.cc:2391] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:10.053673 31066 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:10.059326 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.059398 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:09.988582 1212 raft_consensus.cc:2802] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:10.060083 31071 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:10.045842 31729 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:10.063385 31067 raft_consensus.cc:2408] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:09.992539 993 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:59:10.083256 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.110968 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.111665 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 212791us
I20250902 01:59:10.004791 992 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.111634 31000 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:10.114992 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.131088 30998 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:10.123466 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.112039 30995 raft_consensus.cc:2374] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
W20250902 01:59:10.132553 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.020128 832 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.154599 832 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:10.154919 832 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:10.155243 832 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:10.168491 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 176605us
I20250902 01:59:10.073410 1185 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:09.994194 1027 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:10.170202 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.174319 30995 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:10.170630 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.175160 30997 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.061841 1141 raft_consensus.cc:695] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.170106 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:10.176788 1212 raft_consensus.cc:695] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.176817 1141 consensus_queue.cc:237] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.177747 1212 consensus_queue.cc:237] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:10.180647 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.008239 1263 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.152s
W20250902 01:59:10.170495 30965 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.840s (SENT)
I20250902 01:59:10.073812 1246 raft_consensus.cc:695] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.022782 1024 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:10.186259 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 206287us
W20250902 01:59:10.183635 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.182742 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:10.186218 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.188365 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.188498 31075 raft_consensus.cc:2408] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:10.189908 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:10.192056 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:10.194101 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 226008us
I20250902 01:59:10.197042 30992 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:10.181788 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:10.196422 31695 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.728s (SENT)
I20250902 01:59:10.200960 31695 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.202531 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.203029 993 raft_consensus.cc:695] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.205068 993 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.257037 30997 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:10.259289 31038 leader_election.cc:336] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 1.910s (SENT)
W20250902 01:59:10.264808 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 256766us
I20250902 01:59:10.264395 31038 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.278952 30998 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:10.208076 1270 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.078s
I20250902 01:59:10.208129 978 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:10.257498 1206 raft_consensus.cc:3053] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:10.261986 1185 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.286617 1206 raft_consensus.cc:738] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:10.215097 30936 tablet_copy_service.cc:342] P 4372b4c4c866431faa1bf3e91fd47a88: Request end of tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-4ba5aa5dce214071a172ea365440f0dc received from {username='slave'} at 127.0.0.1:57742
W20250902 01:59:10.287125 1191 consensus_meta.cc:287] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.538s user 0.007s sys 0.003s
W20250902 01:59:10.288446 1260 log.cc:927] Time spent T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20250902 01:59:10.304093 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:10.290623 522 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:10.307916 31075 raft_consensus.cc:1238] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: []
I20250902 01:59:10.309756 31075 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250902 01:59:10.311094 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 119044us
I20250902 01:59:10.292254 1263 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:10.311733 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 108407us
I20250902 01:59:10.298981 1271 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.318307 31066 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 3. Current term is 4. Ops: []
W20250902 01:59:10.355427 30965 leader_election.cc:336] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.846s (SENT)
W20250902 01:59:10.364130 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 54877.4us
W20250902 01:59:10.365316 30967 leader_election.cc:336] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.609s (SENT)
I20250902 01:59:10.365869 30967 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.377871 1276 log.cc:927] Time spent T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20250902 01:59:10.379146 30936 tablet_copy_service.cc:434] P 4372b4c4c866431faa1bf3e91fd47a88: ending tablet copy session 01f6731ce5044a6a948a5ae04da80e6f-4ba5aa5dce214071a172ea365440f0dc on tablet 4ba5aa5dce214071a172ea365440f0dc with peer 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:10.380517 1024 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:10.319680 1263 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:10.368883 1264 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
I20250902 01:59:10.364187 1271 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:10.407213 1271 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:10.379729 1185 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:10.385852 1274 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250902 01:59:10.386055 31038 leader_election.cc:336] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285): Timed out: RequestConsensusVote RPC to 127.29.21.196:44285 timed out after 2.008s (SENT)
I20250902 01:59:10.387842 867 consensus_queue.cc:1046] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:10.388550 1242 raft_consensus.cc:3053] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
W20250902 01:59:10.389070 1267 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20250902 01:59:10.390014 31070 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:10.383726 1273 raft_consensus.cc:2802] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:10.390704 832 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:10.392234 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 91227.6us
I20250902 01:59:10.391669 1206 consensus_queue.cc:260] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.280081 1269 consensus_queue.cc:1046] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.052s
I20250902 01:59:10.305828 1246 consensus_queue.cc:237] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.368644 1027 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:10.404285 30995 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:10.378424 1272 consensus_queue.cc:1046] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250902 01:59:10.410051 31695 leader_election.cc:336] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.875s (SENT)
I20250902 01:59:10.387867 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 910) took 1503 ms. Trace:
W20250902 01:59:10.414091 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 48819.7us
W20250902 01:59:10.416605 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 61233.8us
W20250902 01:59:10.445390 30965 leader_election.cc:336] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.883s (SENT)
W20250902 01:59:10.445586 30967 leader_election.cc:336] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.638s (SENT)
I20250902 01:59:10.412662 1242 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.445909 30965 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.449319 31695 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.449566 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 143189us
I20250902 01:59:10.446200 30967 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.449666 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 84918.3us
I20250902 01:59:10.415320 1273 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.468669 31075 raft_consensus.cc:1238] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250902 01:59:10.470960 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.471362 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.472169 31070 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:10.462405 1281 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:10.474263 31075 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
W20250902 01:59:10.477698 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 83300.5us
I20250902 01:59:10.478055 1284 raft_consensus.cc:3053] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:10.478494 1284 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:10.479665 1248 raft_consensus.cc:3053] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:10.479332 1284 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.480028 1248 raft_consensus.cc:738] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:10.480670 1284 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.480935 1248 consensus_queue.cc:260] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.481829 1241 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:10.470320 1024 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.447504 1206 raft_consensus.cc:3058] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:10.475509 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 106407us
I20250902 01:59:10.475950 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:10.475926 30965 leader_election.cc:336] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.889s (SENT)
I20250902 01:59:10.488039 30965 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.464303 1273 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:10.466272 1147 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.492511 30892 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:10.483968 1263 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.483928 1024 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.496661 1024 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.497145 1242 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:10.498781 1242 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:10.504139 1024 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.504774 31073 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:10.510823 30892 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:10.513200 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.524806 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.505642 1220 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:10.533308 31695 leader_election.cc:336] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.008s (SENT)
W20250902 01:59:10.533887 31692 leader_election.cc:336] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.524s (SENT)
I20250902 01:59:10.535058 31695 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.535247 31692 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.493438 1272 consensus_queue.cc:1046] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
W20250902 01:59:10.543700 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.547149 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.547600 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.549209 1084 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.549655 1084 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.550138 522 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:10.551678 618 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:10.551991 1084 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.554636 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.509094 1281 raft_consensus.cc:3053] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:10.555397 1281 raft_consensus.cc:738] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:10.556218 1281 consensus_queue.cc:260] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.557457 1281 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.559695 1244 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:10.561050 31038 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:10.551465 522 consensus_queue.cc:1046] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20250902 01:59:10.562338 522 raft_consensus.cc:3053] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:10.562680 522 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:10.563714 522 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.564896 522 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.565542 1244 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.565958 1244 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.567833 1244 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.575906 1244 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:10.575994 31038 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:10.576289 1244 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.576680 1244 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.576877 618 raft_consensus.cc:2762] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250902 01:59:10.578433 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:10.579182 618 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250902 01:59:10.579512 618 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:10.579841 618 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:10.586889 618 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.589527 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:10.590263 1084 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:10.590607 1084 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:10.590914 1084 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:10.512665 1271 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:10.512537 1263 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:10.534085 1269 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250902 01:59:10.600769 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 155236us
I20250902 01:59:10.601063 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:10.534755 1220 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.505326 1024 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.606437 31071 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:10.597234 1147 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:10.628767 30967 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.840s (SENT)
I20250902 01:59:10.598101 1191 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:10.600541 1271 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.635041 30967 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.635037 30892 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:10.636130 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 160235us
W20250902 01:59:10.638242 30965 leader_election.cc:336] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.933s (SENT)
W20250902 01:59:10.638125 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 153154us
W20250902 01:59:10.640707 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.444479 31721 rpcz_store.cc:276] 0902 01:59:08.884421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:09.660543 (+776122us) service_pool.cc:225] Handling call
0902 01:59:09.694603 (+ 34060us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:09.713908 (+ 19305us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:09.713937 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:09.714373 (+ 436us) log.cc:844] Serialized 26 byte log entry
0902 01:59:09.835357 (+120984us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:09.839302 (+ 3945us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:09.868366 (+ 29064us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:10.387601 (+519235us) raft_consensus.cc:1683] finished
0902 01:59:10.387633 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:10.387800 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12773,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":110923,"threads_started":1,"wal-append.queue_time_us":195966}
I20250902 01:59:10.639580 1271 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.646744 30926 raft_consensus.cc:1238] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:10.647863 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 121338us
W20250902 01:59:10.648416 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.648877 31692 leader_election.cc:336] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.623s (SENT)
I20250902 01:59:10.649230 31692 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.650641 31692 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:10.651357 31039 leader_election.cc:336] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.706s (SENT)
I20250902 01:59:10.651604 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:10.652379 31721 raft_consensus.cc:2391] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:10.609882 1220 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.664942 1024 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.634183 832 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:10.665756 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.598052 992 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.629009 1286 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:10.667400 992 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.668022 992 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:10.668308 992 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.668596 992 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:10.669025 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 148517us
W20250902 01:59:10.669611 31695 leader_election.cc:336] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.071s (SENT)
W20250902 01:59:10.639322 30965 leader_election.cc:336] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.935s (SENT)
I20250902 01:59:10.670486 30965 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.672406 30967 leader_election.cc:336] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.763s (SENT)
I20250902 01:59:10.673718 31695 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.605166 1244 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.681432 31068 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250902 01:59:10.679482 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.684871 30967 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.673236 1130 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:59:10.699234 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 70581.8us
W20250902 01:59:10.700789 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 50219us
I20250902 01:59:10.675099 991 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:10.689517 1206 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.693667 1191 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:10.690410 618 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:10.721357 1244 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:10.722810 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.765029 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:10.771034 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 55160.4us
I20250902 01:59:10.773799 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:10.774166 31039 leader_election.cc:336] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.601s (SENT)
I20250902 01:59:10.774515 31071 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:10.777014 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:10.777927 31075 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:10.689872 1027 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.779215 30926 raft_consensus.cc:1238] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:10.774561 31039 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.781497 31075 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:10.783893 31038 leader_election.cc:400] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:10.784195 31038 leader_election.cc:403] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250902 01:59:10.784256 31729 raft_consensus.cc:1238] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20250902 01:59:10.786790 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:10.789736 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:10.790310 31728 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:10.791015 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:10.791882 31066 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:10.792354 1263 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.727485 1273 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:10.793756 31721 raft_consensus.cc:3058] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 2 LEARNER]: Advancing to term 4
I20250902 01:59:10.796041 30823 catalog_manager.cc:5582] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 6 to 7, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250902 01:59:10.797034 1291 log.cc:927] Time spent T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20250902 01:59:10.797922 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:10.798719 31723 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:10.800652 1130 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:10.799285 1273 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:10.805713 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 166422us
W20250902 01:59:10.806378 30965 leader_election.cc:336] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.039s (SENT)
I20250902 01:59:10.806721 30965 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.808508 1270 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:10.809079 1270 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:10.793258 1241 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20250902 01:59:10.811874 30965 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.044s (SENT)
I20250902 01:59:10.811077 31730 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:10.818411 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 53505.7us
I20250902 01:59:10.797917 30823 catalog_manager.cc:5582] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:10.819275 31692 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.820670 31692 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:10.823225 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 49098.3us
W20250902 01:59:10.826222 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.744112 1212 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20250902 01:59:10.861200 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 77330.9us
I20250902 01:59:10.785290 1162 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:10.728745 991 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:10.737994 1084 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.875286 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:10.875855 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 206266us
W20250902 01:59:10.866823 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:10.862545 1264 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20250902 01:59:10.878000 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 193259us
I20250902 01:59:10.879669 30967 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:10.876873 31695 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.591s (SENT)
I20250902 01:59:10.880551 31695 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.864789 1244 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:10.871335 978 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:10.864384 1130 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:10.871894 1162 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:10.789238 1027 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:10.891076 31072 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:10.896217 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.891511 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:10.896144 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 75594.3us
I20250902 01:59:10.892172 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:10.892721 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.897828 31692 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.876003 618 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.896978 30994 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:10.896802 31001 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:10.892328 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.876711 1300 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:10.907159 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:10.907325 1303 raft_consensus.cc:2802] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:10.910120 31000 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250902 01:59:10.908428 30965 leader_election.cc:336] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.510s (SENT)
W20250902 01:59:10.912011 30965 leader_election.cc:336] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.189s (SENT)
I20250902 01:59:10.912477 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.915271 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:10.890419 1191 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.919683 30823 catalog_manager.cc:5582] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 5, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:10.884979 1295 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:10.896626 1162 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.922971 31695 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:59:10.928992 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 49437.5us
I20250902 01:59:10.930190 30967 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.931912 30999 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:10.931931 31692 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.922189 1303 raft_consensus.cc:695] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:10.935787 1303 consensus_queue.cc:237] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.939594 1269 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.944561 30993 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:10.951804 30994 raft_consensus.cc:2408] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20250902 01:59:10.954581 30965 leader_election.cc:336] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.207s (SENT)
I20250902 01:59:10.955068 30965 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.958214 31073 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:10.960120 31695 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.927945 1299 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:10.946408 978 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:10.973023 31695 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:10.974272 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 197144us
I20250902 01:59:10.974684 31692 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.975221 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.919780 618 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:10.959827 1286 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.958959 1288 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:10.985147 30967 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:10.984522 31038 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:10.986900 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:10.971802 1174 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:10.995992 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:10.972609 978 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:10.997607 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:10.982656 1248 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:11.014135 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:10.969920 1191 raft_consensus.cc:2802] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:10.993463 832 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250902 01:59:11.020902 1304 log.cc:927] Time spent T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250902 01:59:11.021914 30967 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:11.014616 978 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:10.997684 1174 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:11.027848 1174 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:10.972821 1299 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.031636 1298 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:11.014693 1141 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.032662 1141 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.033030 1299 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:11.033813 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 60858.1us
I20250902 01:59:11.034332 1141 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.034447 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.015599 1292 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:11.035104 1292 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:10.993659 1300 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.020829 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.035432 31695 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:10.994992 1244 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:10.982367 1274 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20250902 01:59:11.018231 1297 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.290s user 0.010s sys 0.010s
I20250902 01:59:11.035415 1292 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.063441 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1675) took 1170 ms. Trace:
I20250902 01:59:11.064354 30967 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:11.017794 1084 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.067880 30997 raft_consensus.cc:1238] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:11.017292 1130 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.067797 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.023485 618 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.019209 991 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.069031 30967 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:11.042148 1303 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:11.070905 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 96334us
I20250902 01:59:11.072521 31068 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:11.112144 31692 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:11.073437 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 75912.5us
I20250902 01:59:11.113260 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:11.114231 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.114707 31001 raft_consensus.cc:2408] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:11.046052 832 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.044940 1244 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:11.076844 30997 raft_consensus.cc:1238] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:11.085352 1220 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.067966 1310 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:11.087944 1027 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.092300 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.095237 1147 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:11.099598 1241 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.101533 1206 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.103969 1320 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.106156 31721 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:59:11.106381 1051 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.108814 992 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.109009 1285 consensus_queue.cc:1046] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:11.067750 31067 rpcz_store.cc:276] 0902 01:59:09.893139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:10.636290 (+743151us) service_pool.cc:225] Handling call
0902 01:59:10.636589 (+ 299us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:10.668836 (+ 32247us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:10.669085 (+ 249us) log.cc:844] Serialized 14 byte log entry
0902 01:59:11.063001 (+393916us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:11.063051 (+ 50us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:11.063208 (+ 157us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:11.063290 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:11.063393 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47551,"prepare.queue_time_us":2389,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":85125,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":336753,"threads_started":2,"wal-append.queue_time_us":221792}
I20250902 01:59:11.072520 1270 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:11.083168 1307 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20250902 01:59:11.129443 31038 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:11.130496 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.111500 618 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.136539 1321 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:11.139441 1130 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:11.143424 30965 leader_election.cc:336] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.343s (SENT)
I20250902 01:59:11.172356 30965 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.144322 30823 catalog_manager.cc:5582] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 4, term changed from 3 to 5, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added. New cstate: current_term: 5 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:11.148036 1147 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.148541 1241 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.149281 31066 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250902 01:59:11.149477 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 80542.3us
W20250902 01:59:11.174819 30965 leader_election.cc:336] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.359s (SENT)
I20250902 01:59:11.128502 1244 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.128276 1185 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:11.177593 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.178692 31001 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:11.178130 1272 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250902 01:59:11.180461 832 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:11.180920 1241 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.178372 1185 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.183899 1185 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.184645 30997 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:11.174789 1310 raft_consensus.cc:695] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.186127 1310 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
W20250902 01:59:11.187719 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 58345us
I20250902 01:59:11.187882 618 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:11.188357 31038 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:11.189285 31038 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:11.190514 1300 raft_consensus.cc:2802] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:11.190591 522 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:11.190980 1300 raft_consensus.cc:695] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.191165 522 raft_consensus.cc:695] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.191862 1300 consensus_queue.cc:237] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.192046 522 consensus_queue.cc:237] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.154074 31692 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.132310 867 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:11.101681 1271 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.134892 1162 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.158269 1317 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:11.158689 1285 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:11.209224 1285 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.156533 1270 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.159449 1174 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.162585 1269 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:11.155988 978 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.164394 992 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.165584 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.215968 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.217582 31067 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.166793 1220 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.171962 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.220892 31071 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.175287 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.152855 1191 raft_consensus.cc:695] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:11.150437 1206 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.188400 618 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.194016 30994 raft_consensus.cc:1273] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250902 01:59:11.151988 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 116633us
I20250902 01:59:11.179842 30823 catalog_manager.cc:5582] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
W20250902 01:59:11.207720 30819 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44316 (request call id 367) took 23110 ms (client timeout 15000 ms). Trace:
I20250902 01:59:11.172966 1321 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.217844 31066 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:11.217716 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 63688.9us
I20250902 01:59:11.176406 1147 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.209955 1271 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.210023 1162 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.213213 1270 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:11.243047 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 68259.6us
I20250902 01:59:11.190534 1292 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.215919 1325 raft_consensus.cc:2802] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election won for term 8
W20250902 01:59:11.242048 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51892.8us
I20250902 01:59:11.240190 31067 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:11.215420 1130 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.258291 31692 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:11.259331 31039 leader_election.cc:336] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.721s (SENT)
I20250902 01:59:11.261474 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.259192 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:11.263343 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.261498 30892 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.264221 1319 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.264681 1319 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.210016 1285 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.271667 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.272560 30997 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:11.248031 1321 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.276777 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.277601 1220 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:11.278093 1220 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.278837 1220 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.253367 867 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:11.274358 1329 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:11.274499 1324 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20250902 01:59:11.274937 1206 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.275112 1325 raft_consensus.cc:695] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:11.275259 1252 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:11.275933 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.265513 1174 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.301219 30997 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:59:11.281978 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.263185 30823 catalog_manager.cc:5582] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 3 to 5, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 5 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
W20250902 01:59:11.289427 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.289705 31066 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:11.305691 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.306396 31730 raft_consensus.cc:2391] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:11.306120 1317 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:11.307228 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.310209 1319 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:11.311303 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.313275 1292 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.319571 1322 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:11.319870 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.320528 31722 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 8
I20250902 01:59:11.323907 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.324509 31724 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:11.325142 1244 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.326030 31073 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:11.326920 1244 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:11.326761 1191 consensus_queue.cc:237] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.296301 1206 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.328039 1206 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.266124 1084 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:11.262867 1320 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.292359 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.294454 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.298558 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.341971 31001 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
W20250902 01:59:11.299250 31039 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.725s (SENT)
I20250902 01:59:11.324142 1325 consensus_queue.cc:237] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.334002 31071 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:11.344715 31038 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.265810 1130 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.353996 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.355355 31038 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.291501 31067 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:59:11.336149 31722 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:11.287842 1162 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:11.344130 31727 raft_consensus.cc:2408] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:11.339334 1084 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.301693 1285 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:11.371613 1084 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:11.374071 30965 leader_election.cc:336] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.606s (SENT)
I20250902 01:59:11.374428 31729 raft_consensus.cc:2408] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:59:11.374482 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.358220 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.375308 31730 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.336031 31721 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:11.348146 30892 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.360100 1252 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:11.363704 1206 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.338879 867 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20250902 01:59:11.368036 1323 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:11.362303 1300 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:11.301362 618 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.369527 1147 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:11.376222 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 77008.3us
W20250902 01:59:11.397037 31039 leader_election.cc:336] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.725s (SENT)
I20250902 01:59:11.358016 30823 catalog_manager.cc:5582] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:11.335003 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.233487 1303 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:11.368311 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.368808 978 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.393802 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.395110 31728 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 3.
I20250902 01:59:11.398787 31066 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.401978 30823 catalog_manager.cc:5582] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 3 to 4, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:11.398686 30892 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.400002 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:11.239423 30819 rpcz_store.cc:269] 0902 01:58:48.063515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:48.069391 (+ 5876us) service_pool.cc:225] Handling call
0902 01:59:11.168420 (+23099029us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:59:06.827636 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:59:06.828177 (+ 541us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:59:06.828231 (+ 54us) write_op.cc:435] Acquired schema lock
0902 01:59:06.828242 (+ 11us) tablet.cc:660] Decoding operations
0902 01:59:06.829512 (+ 1270us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:59:06.829600 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0902 01:59:06.829630 (+ 30us) tablet.cc:683] Acquiring locks for 15 operations
0902 01:59:06.830683 (+ 1053us) tablet.cc:699] Row locks acquired
0902 01:59:06.830702 (+ 19us) write_op.cc:260] PREPARE: finished
0902 01:59:06.830858 (+ 156us) write_op.cc:270] Start()
0902 01:59:06.830989 (+ 131us) write_op.cc:276] Timestamp: P: 1756778346830797 usec, L: 0
0902 01:59:06.831007 (+ 18us) op_driver.cc:348] REPLICATION: starting
0902 01:59:06.871741 (+ 40734us) log.cc:844] Serialized 5925 byte log entry
0902 01:59:08.710174 (+1838433us) op_driver.cc:464] REPLICATION: finished
0902 01:59:09.009427 (+299253us) write_op.cc:301] APPLY: starting
0902 01:59:09.009528 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0902 01:59:09.009934 (+ 406us) tablet.cc:1368] finished BulkCheckPresence
0902 01:59:09.010071 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:59:09.049185 (+ 39114us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b540026f554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212d2ea 00007fc6020ca644 00007fc6020ca5f3 00007fc6020ca568 00007fc6020ca523 00007fc6020c980c 00007fc6020f34a6 00007fc6020db46b 00007fc6020c5acd 00007fc6020c94a5
0902 01:59:09.260267 (+211082us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:59:09.260666 (+ 399us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0902 01:59:09.260746 (+ 80us) write_op.cc:312] APPLY: finished
0902 01:59:09.261216 (+ 470us) log.cc:844] Serialized 107 byte log entry
0902 01:59:09.261397 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:59:09.261686 (+ 289us) write_op.cc:454] Released schema lock
0902 01:59:09.261750 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0902 01:59:09.262229 (+ 479us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:59:09.262264 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3999538,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":196785,"apply.run_cpu_time_us":8922,"apply.run_wall_time_us":252778,"num_ops":15,"prepare.queue_time_us":604029,"prepare.run_cpu_time_us":75998,"prepare.run_wall_time_us":1884046,"raft.queue_time_us":199634,"raft.run_cpu_time_us":18755,"raft.run_wall_time_us":395817,"replication_time_us":1861288,"spinlock_wait_cycles":57596800,"thread_start_us":2120738,"threads_started":4,"wal-append.queue_time_us":1374700,"wal-append.run_cpu_time_us":12759,"wal-append.run_wall_time_us":1843259}]]}
I20250902 01:59:11.404814 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.403836 30921 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:11.407032 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.407346 31693 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.384290 1024 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:11.396260 1300 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:11.380193 1252 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:11.390949 1292 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.411784 1300 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.411942 1252 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.375458 522 consensus_queue.cc:1046] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20250902 01:59:11.413820 1319 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:11.414134 1319 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.414840 1319 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.419101 1185 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.420951 1185 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.421481 1328 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:11.421983 1328 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.423780 1328 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.424546 1303 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.373612 1321 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.428704 1270 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.429199 1321 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.411194 1310 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:11.409679 31723 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:11.431661 1270 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.430130 1310 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.433588 1310 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.434376 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.434656 1310 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.434993 30997 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:11.435187 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.435732 30994 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.396410 1323 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.435882 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.438236 1300 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.438779 1323 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:11.439810 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.440478 30993 raft_consensus.cc:2408] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:11.441133 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:11.442463 30965 leader_election.cc:336] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.822s (SENT)
I20250902 01:59:11.442831 30965 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.443490 1298 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:11.444236 1024 raft_consensus.cc:695] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.444356 1130 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.445076 1024 consensus_queue.cc:237] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.445706 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.446942 1147 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:11.447341 1147 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.447716 1147 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.449937 30823 catalog_manager.cc:5582] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 3, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:11.451263 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 44277us
W20250902 01:59:11.453966 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 97591.3us
I20250902 01:59:11.459408 31730 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:11.460700 30994 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:11.468856 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.481830 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.485544 31728 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:59:11.474361 31039 leader_election.cc:336] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.746s (SENT)
I20250902 01:59:11.447770 1220 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:11.487345 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.488195 31727 raft_consensus.cc:2391] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 3: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:11.487355 30823 catalog_manager.cc:5582] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 3 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:11.393409 1271 consensus_queue.cc:1046] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20250902 01:59:11.493209 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.490568 1244 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20250902 01:59:11.496780 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.490715 1329 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.499876 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.500061 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.500494 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.500813 31038 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.500535 31722 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:11.501175 30921 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:11.502490 31693 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.508188 31068 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:11.507766 1321 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.511713 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.516855 1212 consensus_queue.cc:1046] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:11.512917 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.517665 1212 raft_consensus.cc:3053] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:11.517998 1212 raft_consensus.cc:738] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.518715 1212 consensus_queue.cc:260] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.509699 1130 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:11.519824 1130 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.520131 1130 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.520623 31727 raft_consensus.cc:1238] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20250902 01:59:11.509572 1336 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:11.521518 1319 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:11.486951 1285 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250902 01:59:11.511159 1220 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.479897 1084 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.504422 30823 catalog_manager.cc:5582] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: config changed from index -1 to 2, term changed from 2 to 3, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to <none>, NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) added, no longer has a pending config: VOTER 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193), VOTER 975ecd397f5146409baac87e56a302e9 (127.29.21.194), VOTER 088424002a894fd5bcc11a06af111445 (127.29.21.195), NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } } }
I20250902 01:59:11.529362 1220 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.530736 1300 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.509876 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.534320 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.534857 31727 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.534983 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.535641 30993 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:11.536728 1084 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.537003 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.536738 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.538292 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.538525 30999 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.538808 30993 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:11.509917 1244 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:11.526850 1212 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.537664 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.542079 30992 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:11.542717 31724 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:11.542906 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.539520 1244 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.547052 1255 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:11.545272 31038 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.547108 1244 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.544036 30993 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:11.547658 1336 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.538044 1147 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.549233 1185 raft_consensus.cc:3053] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:11.547250 30995 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:11.553136 1255 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:11.561152 1336 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.561964 1255 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:11.556669 1185 raft_consensus.cc:738] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:11.563503 1185 consensus_queue.cc:260] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.563897 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.564487 1185 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.564934 31038 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:11.565269 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.565654 1329 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:59:11.565755 30918 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.566079 1329 raft_consensus.cc:695] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.567727 1334 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:11.568449 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.568533 1334 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:11.568627 31068 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:11.569231 1334 raft_consensus.cc:695] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:11.570014 1334 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.572109 1147 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.566929 1329 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.572649 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.573241 31722 raft_consensus.cc:2366] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6.
I20250902 01:59:11.573135 30918 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:11.573975 30999 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:11.574160 31693 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:11.574649 31722 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:11.574818 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.575906 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.576936 31068 raft_consensus.cc:2391] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:11.553979 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.578888 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.578703 30823 catalog_manager.cc:5582] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 5 to 7, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:11.581069 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.582011 31725 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:11.583190 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:11.583649 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 309785us
I20250902 01:59:11.584545 31692 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:11.588327 30918 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:11.588415 31039 leader_election.cc:336] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.844s (SENT)
I20250902 01:59:11.586091 30823 catalog_manager.cc:5582] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 2 to 3, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:11.589236 31039 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.590139 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.590643 1084 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:11.590801 31072 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:11.591665 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.583442 1147 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:11.591737 31039 leader_election.cc:336] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.849s (SENT)
I20250902 01:59:11.593206 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.594290 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.595757 31068 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:11.596297 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.597050 30921 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20250902 01:59:11.597323 30965 leader_election.cc:336] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.685s (SENT)
I20250902 01:59:11.597847 30965 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.598006 31069 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250902 01:59:11.598799 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.603295 31066 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:11.614918 1341 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:11.615414 1341 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:11.615806 1341 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.616961 31692 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:11.617660 1323 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:11.614746 30823 catalog_manager.cc:5582] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 reported cstate change: term changed from 1 to 2, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:11.618144 1323 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:11.619194 1323 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.628414 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.632156 31727 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:11.634423 31692 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.641021 31075 raft_consensus.cc:2408] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:59:11.641902 31730 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:59:11.643050 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 45764us
I20250902 01:59:11.652402 31722 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:11.655647 31692 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:11.698670 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.699275 30925 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:11.673583 1323 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:11.673792 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.708700 31724 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:11.710247 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.711297 31724 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.712656 1310 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:59:11.714994 1300 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250902 01:59:11.640302 1252 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.717206 1252 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:11.718056 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.718717 31069 raft_consensus.cc:2391] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:11.720083 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.652539 1298 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250902 01:59:11.721441 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.690941 1292 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:11.724170 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.724929 30925 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:11.725349 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.724224 1292 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:11.725994 31074 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:11.726780 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.727237 1292 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:11.728353 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.729346 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.729700 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.730368 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.730705 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.731418 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.732676 30923 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:11.657568 31039 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.910s (SENT)
I20250902 01:59:11.652760 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.592594 1147 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:11.659519 31729 raft_consensus.cc:1238] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
W20250902 01:59:11.666265 30965 leader_election.cc:336] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.121s (SENT)
I20250902 01:59:11.672454 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.674481 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.675593 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.681452 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.686833 31727 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:11.688277 31072 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:11.692513 31695 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
W20250902 01:59:11.694782 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.725229 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.939522 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:11.940392 31075 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:11.733783 31068 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:11.731390 30919 raft_consensus.cc:2391] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:11.739748 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.941892 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.942168 31730 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:11.929667 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:11.742177 30924 raft_consensus.cc:1238] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 2. Current term is 3. Ops: []
I20250902 01:59:11.943001 31069 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:59:11.819665 31725 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:11.945879 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.946568 31725 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:11.946951 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:11.948778 31726 raft_consensus.cc:1273] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:11.942536 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.949445 30918 raft_consensus.cc:2391] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:11.949937 31722 raft_consensus.cc:2391] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:11.950413 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 257990us
I20250902 01:59:11.735664 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:11.868762 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:11.731541 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.943222 30965 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:11.942497 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 286984us
I20250902 01:59:11.942162 30920 raft_consensus.cc:2366] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:11.740061 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.951201 31695 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.974560 30964 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:11.962304 30923 raft_consensus.cc:2366] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:11.959558 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:11.976025 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.775148 1317 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250902 01:59:11.977102 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.005281 31727 raft_consensus.cc:3058] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:11.989297 31001 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:11.978832 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.007184 30923 raft_consensus.cc:2366] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:12.007675 31074 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.007824 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:11.984030 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.986249 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:11.988615 30993 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:59:12.008535 30924 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:12.008708 30999 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:12.008917 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.009485 30994 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.010442 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.003557 30922 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.011104 30918 raft_consensus.cc:2391] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:12.011602 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.012043 31725 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:12.013068 31038 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:12.013684 30926 raft_consensus.cc:2391] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.015511 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:12.016050 30997 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:12.016981 31692 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:12.017359 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:12.017580 31072 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:59:12.021278 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.022539 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:11.771402 1212 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:12.024403 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 358218us
I20250902 01:59:11.719570 1321 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.025885 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.018288 30925 raft_consensus.cc:2374] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4. Re-sending same reply.
I20250902 01:59:11.981173 1325 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:12.031865 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.035992 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.033063 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 375522us
I20250902 01:59:11.771615 1130 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:11.771756 31723 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:12.037707 31727 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:12.042341 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.017793 1336 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:12.009994 1337 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.036624 30922 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:12.010579 1292 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:12.040526 30917 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.041410 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.092697 30919 raft_consensus.cc:2391] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:11.973520 992 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20250902 01:59:12.042824 31038 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:12.038192 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.093835 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.044477 31725 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:12.095611 30964 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:12.025379 1317 consensus_queue.cc:1046] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20250902 01:59:12.102581 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:11.862185 1341 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:12.048406 31039 leader_election.cc:336] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.000s (SENT)
I20250902 01:59:12.049270 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.108170 30924 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:12.068290 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.069006 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.044466 1212 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.069689 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:12.070181 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.111402 30922 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.022763 1255 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.072013 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.122074 30920 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.036209 1191 consensus_queue.cc:1046] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20250902 01:59:12.076786 30994 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:12.112929 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:12.124209 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.078045 31728 raft_consensus.cc:2366] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7.
I20250902 01:59:12.082226 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.025883 1298 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:12.017567 1244 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.142724 31727 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEARNER]: Advancing to term 6
I20250902 01:59:12.105667 1337 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:12.099623 1270 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:12.099633 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.093694 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.046677 1147 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.051458 1325 raft_consensus.cc:695] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:12.112247 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 63866.4us
I20250902 01:59:12.151115 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.128834 1130 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:12.158392 1341 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.131103 1212 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:12.131654 31723 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:12.159335 1325 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.160271 31721 raft_consensus.cc:1273] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:12.132172 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.163971 30926 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:12.132923 30922 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 6. Ops: []
W20250902 01:59:12.133368 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 455372us
I20250902 01:59:12.094544 1336 raft_consensus.cc:695] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:12.044723 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.097689 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 54999.8us
I20250902 01:59:12.146651 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.182008 30994 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:12.182248 30993 raft_consensus.cc:1273] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:12.148900 31725 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 6. Current term is 7. Ops: [5.7-6.8]
I20250902 01:59:12.106321 1252 raft_consensus.cc:2802] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:12.183496 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:12.123873 1317 raft_consensus.cc:3053] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:12.146250 1337 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:12.154865 1350 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250902 01:59:12.230863 30994 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:12.156121 1255 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:12.157316 31073 raft_consensus.cc:2366] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:59:12.128729 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.232182 30923 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.168803 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.232399 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:12.232637 30999 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:12.232944 30992 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:12.127553 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.235998 30919 raft_consensus.cc:2391] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.164987 30924 raft_consensus.cc:3053] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Stepping down as leader of term 7
W20250902 01:59:12.184578 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 233475us
I20250902 01:59:12.186842 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.142661 1292 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250902 01:59:12.188199 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.194008 31074 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
W20250902 01:59:12.197950 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:12.155491 1244 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:12.231197 31727 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:12.274019 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:12.275089 30965 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.239s (SENT)
I20250902 01:59:12.248853 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.186686 1317 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:12.227073 1298 consensus_queue.cc:1035] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.145s
I20250902 01:59:12.515007 30997 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:12.515380 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 364213us
I20250902 01:59:12.281941 30967 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:12.519706 31039 leader_election.cc:336] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.072s (SENT)
W20250902 01:59:12.520061 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 238230us
W20250902 01:59:12.520486 31039 leader_election.cc:336] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.074s (SENT)
I20250902 01:59:12.520987 30967 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:12.521292 31039 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.101s (SENT)
I20250902 01:59:12.521759 31039 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.448875 1323 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:12.522969 1359 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.523859 1084 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.524276 1084 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:12.340081 30921 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:12.524878 31039 leader_election.cc:336] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.171s (SENT)
I20250902 01:59:12.525292 31039 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:12.358742 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:12.503510 1372 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.115s user 0.001s sys 0.001s
I20250902 01:59:12.517916 31695 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.525947 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.526494 1084 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:12.526880 30921 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:12.405448 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.528266 30917 raft_consensus.cc:2408] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20250902 01:59:12.508263 31070 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.006s sys 0.017s
W20250902 01:59:12.386507 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 372906us
W20250902 01:59:12.529356 31070 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.788s user 0.006s sys 0.017s
I20250902 01:59:12.530876 1141 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.249719 31723 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:12.531358 1141 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:12.329381 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.533152 1141 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.534685 31071 raft_consensus.cc:1273] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250902 01:59:12.361429 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.536068 30918 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.536895 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.538182 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.226765 1337 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.522953 1323 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.526516 1341 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:12.527441 1319 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.540599 1341 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:12.540958 1341 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:12.342336 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.544575 31000 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 4: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:12.366533 31728 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:59:12.276472 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:12.547497 1319 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.518438 30965 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.548060 1341 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.548781 30999 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
W20250902 01:59:12.549793 1285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.022s sys 0.018s
W20250902 01:59:12.550120 1285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s user 0.022s sys 0.018s
I20250902 01:59:12.477701 30926 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
W20250902 01:59:12.548853 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 273824us
I20250902 01:59:12.323899 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:12.551323 30965 leader_election.cc:336] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.931s (SENT)
I20250902 01:59:12.551671 31726 raft_consensus.cc:2366] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5.
W20250902 01:59:12.372238 1342 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20250902 01:59:12.276173 30992 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250902 01:59:12.552711 1323 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.276074 30924 raft_consensus.cc:738] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:12.557561 31693 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:12.528635 31695 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.553741 1366 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20250902 01:59:12.558362 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:12.406955 1336 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.559275 1270 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:12.560364 1270 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.367851 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.343809 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:12.562391 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.528504 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:12.329200 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 48253.7us
I20250902 01:59:12.563071 30920 raft_consensus.cc:2391] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.376423 867 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.563704 867 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.564267 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.559353 1376 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:12.338515 31724 consensus_meta.cc:287] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.626s user 0.005s sys 0.003s
I20250902 01:59:12.563750 1084 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250902 01:59:12.528534 31039 leader_election.cc:336] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.206s (SENT)
I20250902 01:59:12.343700 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.557072 30924 consensus_queue.cc:260] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.572547 31039 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.565128 30921 raft_consensus.cc:2391] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:12.565438 30965 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.155s (SENT)
I20250902 01:59:12.573457 30965 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.573940 30997 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:12.504258 1364 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:12.567209 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 402977us
I20250902 01:59:12.567792 1191 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:12.567346 1303 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:12.273309 1252 raft_consensus.cc:695] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:12.568125 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:12.594673 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 66151.4us
I20250902 01:59:12.551456 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:12.595719 31039 leader_election.cc:336] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.245s (SENT)
I20250902 01:59:12.583401 1292 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.596242 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:12.597034 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:12.599164 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:12.599664 30997 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:12.603257 30926 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:12.570084 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:12.625649 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 55627.6us
I20250902 01:59:12.629251 31727 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:12.553966 1317 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:12.507088 1373 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.117s user 0.001s sys 0.001s
W20250902 01:59:12.461367 1354 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.722s user 0.008s sys 0.000s
I20250902 01:59:12.564893 30922 raft_consensus.cc:2366] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:12.565537 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.577456 30992 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:12.616845 1244 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.580291 31073 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1683) took 1001 ms. Trace:
I20250902 01:59:12.635963 31073 rpcz_store.cc:276] 0902 01:59:11.578457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:11.578614 (+ 157us) service_pool.cc:225] Handling call
0902 01:59:12.580084 (+1001470us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b5400157054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:12.580230 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1328549760}
I20250902 01:59:12.635072 1317 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.463332 1328 raft_consensus.cc:3053] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:12.584411 30999 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:12.637075 1328 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:12.584803 31001 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:12.637724 31726 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:12.637969 1328 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.639020 1328 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.641561 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.633314 1298 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20250902 01:59:12.644656 1298 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:12.644501 1329 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:59:12.644977 1298 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:12.550447 1341 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.604393 1376 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:12.604458 31000 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20250902 01:59:12.605674 867 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:12.645753 1298 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.647020 1298 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:12.606272 30924 raft_consensus.cc:3058] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:12.606737 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 50788.4us
I20250902 01:59:12.650794 1252 consensus_queue.cc:237] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.607733 30995 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:12.655426 1285 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250902 01:59:12.617029 1303 raft_consensus.cc:695] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:12.656169 1285 consensus_queue.cc:1230] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250902 01:59:12.656922 1303 consensus_queue.cc:237] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.608345 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:12.610993 1337 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.615343 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.658874 30998 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:12.659837 1337 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.616240 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.662134 30921 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:12.662899 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.663455 30921 raft_consensus.cc:2366] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6.
I20250902 01:59:12.664100 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.664741 30921 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:12.616395 31039 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.335s (SENT)
I20250902 01:59:12.618870 1386 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:12.625241 1375 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.610149 30997 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:12.622112 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.572448 31724 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:12.636984 1321 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250902 01:59:12.611532 1319 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:12.631577 1336 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:12.632669 1359 raft_consensus.cc:1062] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:59:12.539333 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:12.634294 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 68843.4us
I20250902 01:59:12.635646 31038 leader_election.cc:400] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5.
I20250902 01:59:12.587023 30923 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.611665 1364 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.661612 31725 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:12.665544 31039 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:12.680176 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 63810us
W20250902 01:59:12.680678 31039 leader_election.cc:336] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.477s (SENT)
I20250902 01:59:12.680960 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.682497 31066 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
W20250902 01:59:12.682927 31039 leader_election.cc:336] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.541s (SENT)
I20250902 01:59:12.683146 31066 raft_consensus.cc:604] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.683689 31066 raft_consensus.cc:612] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:12.684116 1321 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.665963 1375 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:12.686874 1375 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:12.666247 1378 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20250902 01:59:12.670624 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.665362 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.683305 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.688455 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.670917 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.704360 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.705199 30923 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.690202 30920 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:12.708978 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.709813 30920 raft_consensus.cc:2391] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.677397 31695 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:12.677529 31038 leader_election.cc:403] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20250902 01:59:12.618768 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 53357us
I20250902 01:59:12.675238 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:12.696610 30922 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:12.712337 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.710565 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 188442us
I20250902 01:59:12.736732 30919 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:12.737617 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:12.738986 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 103489us
I20250902 01:59:12.683847 1319 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.746824 31066 pending_rounds.cc:85] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Aborting all ops after (but not including) 1
I20250902 01:59:12.747229 31066 pending_rounds.cc:107] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250902 01:59:12.747514 31066 raft_consensus.cc:2887] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:12.749454 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:12.690521 1147 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:59:12.672821 1359 raft_consensus.cc:1066] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Unable to promote non-voter 01f6731ce5044a6a948a5ae04da80e6f: Illegal state: Leader has not yet committed an operation in its own term
I20250902 01:59:12.679320 1364 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:12.672528 1336 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:12.764637 1329 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:12.774919 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.740242 1325 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.793978 1329 raft_consensus.cc:695] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:12.776331 1375 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.795282 1329 consensus_queue.cc:237] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.777577 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.783818 30921 raft_consensus.cc:2366] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 6. Current term is 7.
I20250902 01:59:12.765002 522 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.042s
I20250902 01:59:12.812979 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1691) took 1219 ms. Trace:
I20250902 01:59:12.781299 1285 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.816360 1285 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:12.746338 1244 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:12.792758 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.745872 1084 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20250902 01:59:12.741544 1321 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:12.797187 1336 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:12.822496 1380 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.086s user 0.002s sys 0.000s
I20250902 01:59:12.813472 31070 rpcz_store.cc:276] 0902 01:59:11.593847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:11.593950 (+ 103us) service_pool.cc:225] Handling call
0902 01:59:11.600657 (+ 6707us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:11.601020 (+ 363us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:11.741518 (+140498us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b180036b140. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff14665a 00007fc608617409 00007fc5ff007af0 00007fc5ff1cb894 00007fc5ff1cb805 00007fc605b15327 00007fc60213a957 00007fc60214a06f 00007fc602149fe8
0902 01:59:12.529718 (+788200us) log.cc:844] Serialized 14 byte log entry
0902 01:59:12.616464 (+ 86746us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:12.620635 (+ 4171us) log.cc:844] Serialized 26 byte log entry
0902 01:59:12.620760 (+ 125us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:12.620795 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:12.620873 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:12.812634 (+191761us) raft_consensus.cc:1683] finished
0902 01:59:12.812756 (+ 122us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:12.812910 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70878,"prepare.queue_time_us":174107,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":28686,"raft.queue_time_us":35767,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":21193,"spinlock_wait_cycles":25597824,"thread_start_us":854892,"threads_started":2,"wal-append.queue_time_us":89784}
I20250902 01:59:12.820089 30920 raft_consensus.cc:2408] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:12.844445 30924 raft_consensus.cc:1273] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:12.845876 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.846531 30924 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
W20250902 01:59:12.896121 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:12.904361 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:12.907797 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 224901us
W20250902 01:59:12.908999 31039 leader_election.cc:336] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.696s (SENT)
W20250902 01:59:12.909943 31039 leader_election.cc:336] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.757s (SENT)
W20250902 01:59:12.911278 31039 leader_election.cc:336] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.648s (SENT)
I20250902 01:59:12.913367 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.915022 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 70640.2us
I20250902 01:59:12.915582 30926 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:12.916559 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.916821 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.919063 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 165094us
I20250902 01:59:12.920241 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:12.919426 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 242045us
I20250902 01:59:12.927644 1384 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.928120 1384 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:12.919050 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 298013us
I20250902 01:59:12.922365 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.929507 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.929994 30925 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:12.930433 30923 raft_consensus.cc:2374] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7. Re-sending same reply.
I20250902 01:59:12.931078 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.932721 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.933151 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.933286 30926 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:12.933651 30922 raft_consensus.cc:2391] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:12.936561 31039 leader_election.cc:336] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.081s (SENT)
I20250902 01:59:12.936721 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:12.936916 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:12.937088 31039 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:12.937278 31069 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:12.937439 31073 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:12.940310 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.940703 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.940819 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.941640 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:12.942236 30924 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.940886 30919 raft_consensus.cc:2391] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:12.942682 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.938513 30889 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:12.951743 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:12.952493 31073 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:12.961184 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:12.961783 31067 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250902 01:59:12.962021 31074 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:12.963400 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:12.972218 31695 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:12.943642 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.934427 1024 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.930871 1321 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250902 01:59:12.989663 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:12.935822 1337 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.920734 1298 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250902 01:59:12.942358 1384 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:12.942299 1368 consensus_queue.cc:1046] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
W20250902 01:59:13.009248 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:13.025154 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.014638 1387 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20250902 01:59:12.991672 30967 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:12.951088 1381 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.020212 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:13.020653 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:13.022202 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.023474 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 86939.4us
I20250902 01:59:12.921180 1366 consensus_queue.cc:1046] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20250902 01:59:12.964571 1252 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:13.025352 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.943876 1386 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:13.028563 30967 leader_election.cc:400] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:59:12.942857 1364 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20250902 01:59:12.986488 1141 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250902 01:59:13.033326 30917 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:13.038622 30998 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20250902 01:59:12.993196 1325 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.037384 1381 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.040694 30997 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:13.034721 30995 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:13.002660 1147 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:13.037009 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.037676 30923 raft_consensus.cc:2391] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:13.046253 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:12.938860 1336 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.047123 30923 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:13.025394 1384 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:12.990006 1376 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.051s
I20250902 01:59:13.034590 867 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:13.005131 1024 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.055029 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.068284 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.095882 31722 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:13.069643 31692 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:13.041796 1375 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.026016 1337 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.101814 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.102736 30923 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:13.076468 30889 leader_election.cc:400] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6.
I20250902 01:59:13.103327 30889 leader_election.cc:403] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250902 01:59:13.104151 30964 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:13.050369 1388 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.025874 1317 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250902 01:59:13.077131 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.114092 30919 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:13.094218 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.090406 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 59852.3us
I20250902 01:59:13.048799 1147 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:13.114084 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.106694 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 84438.9us
I20250902 01:59:13.051398 1386 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:13.118378 1386 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.119256 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.120678 1386 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.121511 1386 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:13.121584 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.122030 1386 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:13.122498 1386 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:13.123060 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.120682 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.124373 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.124959 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.115044 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 192099us
I20250902 01:59:13.125844 30922 raft_consensus.cc:2366] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 6. Current term is 7.
W20250902 01:59:13.126250 30965 leader_election.cc:336] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.187s (SENT)
I20250902 01:59:13.126772 30965 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:13.128705 30965 leader_election.cc:336] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.826s (SENT)
I20250902 01:59:13.129477 30965 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.039803 1325 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.127225 30922 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250902 01:59:13.131634 1325 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.132323 1325 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:13.115216 31074 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:59:13.133708 31074 raft_consensus.cc:604] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.132823 1325 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.125846 30925 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:13.136963 30925 raft_consensus.cc:3053] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:13.137431 30925 raft_consensus.cc:738] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:13.137920 1325 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:13.138445 30925 consensus_queue.cc:260] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.035718 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.140280 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.140496 31695 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.117030 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.142279 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:13.143151 30923 raft_consensus.cc:1273] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:13.048955 1363 consensus_queue.cc:1035] T 349593d2d3574cd6ab281c105756f435 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20250902 01:59:13.143736 1325 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:13.173565 31727 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:13.093874 1084 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:13.144933 1402 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:13.132803 30922 raft_consensus.cc:1273] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:13.052273 1379 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20250902 01:59:13.143290 30924 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:13.198253 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:13.147437 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.199739 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.191443 31039 leader_election.cc:336] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.603s (SENT)
I20250902 01:59:13.195088 30923 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:13.201362 30923 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:13.049474 1191 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20250902 01:59:13.218637 30924 raft_consensus.cc:1273] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250902 01:59:13.110616 1341 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:13.223217 30918 raft_consensus.cc:1273] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:59:13.198109 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:13.109817 1252 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:13.202526 31039 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.895s (SENT)
I20250902 01:59:13.174449 1147 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:13.228325 31722 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:13.205132 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:13.228902 30917 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:13.200310 30919 raft_consensus.cc:2374] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:13.191273 1329 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.094791 1384 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.227216 31039 leader_election.cc:336] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.994s (SENT)
I20250902 01:59:13.203066 30922 raft_consensus.cc:1273] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:13.112015 1375 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.115590 1130 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20250902 01:59:13.277699 30994 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250902 01:59:13.280844 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 152184us
I20250902 01:59:13.226703 1376 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:13.287137 30994 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:13.231240 1341 raft_consensus.cc:695] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:13.223472 1255 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20250902 01:59:13.202073 1084 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.296113 31069 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:13.313503 30925 raft_consensus.cc:3058] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:13.320093 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.271302 1402 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:13.231449 1303 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20250902 01:59:13.231701 1252 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.116526 1388 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.333986 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 124620us
I20250902 01:59:13.335207 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:13.295601 1292 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:13.341070 30918 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250902 01:59:13.342686 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.329093 31039 leader_election.cc:336] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.223s (SENT)
I20250902 01:59:13.343479 30922 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:13.307718 1319 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:13.344791 31693 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:13.332583 1392 log.cc:927] Time spent T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
W20250902 01:59:13.338056 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 308602us
I20250902 01:59:13.280481 1329 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.322919 1366 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6.
I20250902 01:59:13.343451 31039 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.326252 1325 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20250902 01:59:13.303216 1376 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.357599 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:13.324429 1379 consensus_queue.cc:1035] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20250902 01:59:13.362428 31695 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.364382 1252 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.350742 1319 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.367937 31074 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250902 01:59:13.351682 1317 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.347281 1336 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:13.388572 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 59539.2us
I20250902 01:59:13.388713 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:13.389505 31039 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.626s (SENT)
I20250902 01:59:13.358088 1024 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:13.390782 31039 leader_election.cc:336] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.698s (SENT)
W20250902 01:59:13.392482 31039 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.489s (SENT) [suppressed 89 similar messages]
I20250902 01:59:13.341572 1400 consensus_queue.cc:1035] T 331364bf121c480483b11ff32993e1ec P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20250902 01:59:13.393301 30889 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:13.395828 31067 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:59:13.397060 31067 raft_consensus.cc:604] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.397446 31067 raft_consensus.cc:612] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:13.346128 1292 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:13.365881 1337 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:13.441825 30965 leader_election.cc:336] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.132s (SENT)
I20250902 01:59:13.442425 30965 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.388867 1381 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:13.450615 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 57407.1us
I20250902 01:59:13.454990 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:13.365106 1272 consensus_queue.cc:1046] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
W20250902 01:59:13.450381 1391 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
W20250902 01:59:13.488674 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 46903.9us
I20250902 01:59:13.400246 1415 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:13.490357 1272 raft_consensus.cc:3053] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:13.490751 1272 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:13.456698 1024 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:13.491814 1393 log.cc:927] Time spent T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20250902 01:59:13.491914 1024 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:13.491641 1272 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.492846 1024 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.493690 1399 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250902 01:59:13.493994 1415 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:13.494679 1390 log.cc:927] Time spent T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20250902 01:59:13.494900 1415 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.451186 1406 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.421346 1417 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:13.360189 1341 consensus_queue.cc:237] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.500000 1396 log.cc:927] Time spent T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20250902 01:59:13.384383 522 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:13.466382 30889 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:13.499892 1328 consensus_meta.cc:287] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.861s user 0.013s sys 0.000s
I20250902 01:59:13.466403 1285 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.502102 1417 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:13.521633 1329 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.457558 1255 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:13.523093 1376 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.503669 1414 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:13.523249 1386 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.521631 1417 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:13.525197 1386 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.504273 1406 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.527752 1411 consensus_queue.cc:1035] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250902 01:59:13.528400 1406 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:13.498684 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.504926 1395 log.cc:927] Time spent T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20250902 01:59:13.497859 1325 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:13.531234 1325 raft_consensus.cc:695] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:13.532289 1325 consensus_queue.cc:237] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.505718 1403 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20250902 01:59:13.505972 1244 consensus_meta.cc:287] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.687s user 0.009s sys 0.000s
I20250902 01:59:13.538071 1244 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5.
I20250902 01:59:13.511485 1292 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.512395 1319 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.518396 522 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.545701 522 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.513953 1381 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:13.545948 31070 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:13.465883 1408 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20250902 01:59:13.546463 31071 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "416020230983423c9f843070a7893658"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
I20250902 01:59:13.487761 1368 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:13.547173 31071 raft_consensus.cc:604] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.392091 1317 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.520777 1388 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:13.501960 1375 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.547642 31071 raft_consensus.cc:612] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:13.537482 1285 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election won for term 3
I20250902 01:59:13.453181 1337 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250902 01:59:13.549183 1285 raft_consensus.cc:695] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:13.509102 1084 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:13.536365 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 70100.1us
I20250902 01:59:13.535319 30925 raft_consensus.cc:1273] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250902 01:59:13.509954 1272 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:13.548030 1147 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.555678 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:13.556712 1147 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.550573 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1921) took 1056 ms. Trace:
I20250902 01:59:13.558952 30925 rpcz_store.cc:276] 0902 01:59:12.494263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.136484 (+642221us) service_pool.cc:225] Handling call
0902 01:59:13.136815 (+ 331us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:13.550365 (+413550us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:13.550528 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:13.550204 1285 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.551265 1368 consensus_queue.cc:1035] T ede06f18c6574c57857475151e3767d2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250902 01:59:13.552027 30889 leader_election.cc:400] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7.
I20250902 01:59:13.560917 30889 leader_election.cc:403] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250902 01:59:13.552266 1317 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:13.561575 1408 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7.
W20250902 01:59:13.550733 30965 leader_election.cc:336] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.185s (SENT)
I20250902 01:59:13.566722 30965 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.567420 1375 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:13.571131 1298 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:13.571583 1298 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:13.572381 1298 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.575471 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:13.576082 1424 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.576653 1424 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.577314 1389 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.353s user 0.009s sys 0.000s
I20250902 01:59:13.578724 1424 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:13.615906 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.638980 31039 leader_election.cc:336] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.170s (SENT)
I20250902 01:59:13.639635 31039 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.670425 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:13.676980 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.788692 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 149766us
W20250902 01:59:13.792894 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 122602us
I20250902 01:59:13.794253 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:13.797178 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.797582 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.798249 30889 connection.cc:232] RPC call timeout handler was delayed by -1.1342195750000883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20250902 01:59:13.801451 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:13.802050 30923 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.007s sys 0.032s
W20250902 01:59:13.802995 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 412818us
W20250902 01:59:13.803915 30923 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.542s user 0.007s sys 0.032s
I20250902 01:59:13.806463 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:13.807017 31725 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:13.808595 31722 raft_consensus.cc:1273] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:13.812930 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:13.813895 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1723) took 1041 ms. Trace:
I20250902 01:59:13.814275 30920 rpcz_store.cc:276] 0902 01:59:12.772395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.012575 (+240180us) service_pool.cc:225] Handling call
0902 01:59:13.812729 (+800154us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b5800000f98. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:13.813825 (+ 1096us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4152576}
I20250902 01:59:13.788142 1317 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.821430 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:13.818030 31728 raft_consensus.cc:1238] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:13.810214 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.792323 1147 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.796512 1424 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:13.827863 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.799508 1388 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:13.797127 1381 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:13.799862 1298 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:13.828521 31726 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:13.832899 30889 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:13.800101 1323 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.020s sys 0.020s
I20250902 01:59:13.834127 31066 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:13.834410 31725 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
W20250902 01:59:13.873349 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:13.868005 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.010s sys 0.018s
I20250902 01:59:13.868077 31727 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:13.874011 31730 raft_consensus.cc:1273] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20250902 01:59:13.875216 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.835s user 0.010s sys 0.018s
I20250902 01:59:13.581148 1417 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.879667 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 517607us
I20250902 01:59:13.826233 1366 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250902 01:59:13.878038 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 57055.6us
W20250902 01:59:13.887691 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 54895.1us
I20250902 01:59:13.888661 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:13.816763 1292 consensus_queue.cc:1035] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.071s
I20250902 01:59:13.891218 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.891608 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:13.891796 30924 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.002s sys 0.029s
W20250902 01:59:13.875823 1323 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.020s sys 0.020s
I20250902 01:59:13.899782 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:13.882905 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.006s sys 0.023s
W20250902 01:59:13.901923 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s user 0.006s sys 0.023s
W20250902 01:59:13.902292 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 798269us
I20250902 01:59:13.905983 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.907019 31001 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20250902 01:59:13.908064 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.908839 30999 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:13.910830 30999 raft_consensus.cc:1273] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250902 01:59:13.895860 30924 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.616s user 0.002s sys 0.029s
I20250902 01:59:13.819708 1084 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20250902 01:59:13.896332 30995 raft_consensus.cc:1273] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250902 01:59:13.897696 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.916857 30994 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:13.898662 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.917870 30993 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:13.899046 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:13.920691 31692 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.924221 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:13.924916 30997 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 6
I20250902 01:59:13.829694 1317 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:13.893622 1328 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:13.934881 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.016s sys 0.085s
I20250902 01:59:13.935104 31725 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250902 01:59:13.938292 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 46208.7us
I20250902 01:59:14.029006 31039 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:13.832264 1147 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:13.901541 31723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.504s user 0.016s sys 0.043s
I20250902 01:59:13.891147 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:13.882269 1368 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:13.988830 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1731) took 1213 ms. Trace:
W20250902 01:59:13.923791 867 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.008s sys 0.029s
I20250902 01:59:13.981550 31000 raft_consensus.cc:1273] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:59:13.997447 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:13.966850 31067 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:14.099375 30995 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:13.982010 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:14.094920 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 64381.9us
I20250902 01:59:14.101401 31695 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:13.896209 1325 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:13.997869 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.006232 30997 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:14.010555 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.102783 31001 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:14.103193 31726 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:13.882184 1244 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20250902 01:59:13.914470 1375 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:14.067808 1364 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:14.029943 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.018s sys 0.086s
I20250902 01:59:13.928953 1406 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:14.107429 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.038986 31729 raft_consensus.cc:2408] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:14.025000 30964 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:14.056033 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:14.067605 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 179028us
I20250902 01:59:14.071305 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.135030 30889 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:14.135486 31725 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:14.136243 30889 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:14.136483 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.137305 31725 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:14.138237 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.076038 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.104815 31038 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:14.108268 30997 raft_consensus.cc:1273] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250902 01:59:14.108858 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 79948.8us
I20250902 01:59:14.140267 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.141619 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.142345 31726 raft_consensus.cc:2391] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:14.142540 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 117625us
I20250902 01:59:14.144176 31692 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:14.145985 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:14.146930 30996 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:14.141256 30889 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:14.144241 30964 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:14.139387 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.152237 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:14.152773 31000 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:14.009980 1328 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.017529 1421 consensus_queue.cc:1046] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.235s
W20250902 01:59:14.143245 1272 consensus_meta.cc:287] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.590s user 0.008s sys 0.000s
I20250902 01:59:14.131978 31001 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:14.149616 31721 raft_consensus.cc:2366] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6.
I20250902 01:59:14.162366 30995 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:14.166656 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:14.157187 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:14.182791 31725 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:14.105711 1405 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.169869 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.186110 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.186525 31726 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
W20250902 01:59:14.161176 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 56451.6us
I20250902 01:59:14.142859 1364 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:14.187850 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:14.098356 867 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.982s user 0.008s sys 0.037s
I20250902 01:59:14.094517 1411 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.190454 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:14.190697 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.190904 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.191037 30992 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:14.149381 1292 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.192001 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.196775 30996 raft_consensus.cc:2408] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:14.208469 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:14.208635 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.155881 1359 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.179147 1084 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:14.209481 31000 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:14.186432 1381 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:14.104846 1329 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.102s
W20250902 01:59:14.237632 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 93571us
I20250902 01:59:14.277161 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:14.196563 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 507) took 1189 ms. Trace:
I20250902 01:59:14.191157 1364 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:14.153023 1406 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.191541 867 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:14.131055 1325 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.298539 31725 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:14.299361 30992 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:14.311118 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:14.276518 1382 log.cc:927] Time spent T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250902 01:59:14.302085 30964 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:14.317164 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 100838us
I20250902 01:59:14.152185 1375 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:14.311686 30992 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:14.318482 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:14.300326 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:14.197764 1255 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20250902 01:59:14.233311 1084 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:14.281260 1321 raft_consensus.cc:3053] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Stepping down as leader of term 2
I20250902 01:59:14.209254 1292 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.334954 31725 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:14.347982 30995 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:14.353596 30992 raft_consensus.cc:1273] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:14.355311 31726 raft_consensus.cc:2391] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:14.356379 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 168615us
I20250902 01:59:14.317895 30964 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:14.360127 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 190274us
I20250902 01:59:14.303934 1426 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:14.361753 30996 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:14.333871 1402 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:14.340889 1084 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:14.206871 1411 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:14.383244 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 65968.1us
I20250902 01:59:14.342471 1321 raft_consensus.cc:738] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:14.200958 1405 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.399503 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:14.404111 31067 raft_consensus.cc:2408] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20250902 01:59:14.404222 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 85823us
I20250902 01:59:14.400539 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:14.400772 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:14.232131 1359 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.355849 1417 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:14.351846 1244 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.400087 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:14.370317 1141 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:14.386458 1427 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.397025 1328 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:14.405597 31075 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:14.409845 30996 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250902 01:59:14.378201 1408 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:14.430864 1408 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:14.421851 31073 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:14.376271 1317 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:14.398171 1271 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:14.397655 31721 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:14.401695 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:14.432523 30919 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 6
I20250902 01:59:14.432796 1317 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:14.433116 31721 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:14.433457 1317 consensus_queue.cc:1035] T 24426e817ea94ac5a7014c3457ea087c P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:14.434697 1317 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.388118 1402 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.435775 1402 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:14.408339 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:14.408530 1252 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.002s sys 0.002s
I20250902 01:59:14.424705 1428 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:14.439777 1252 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.002s sys 0.002s
I20250902 01:59:14.439857 30917 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:14.439944 1428 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.440510 1252 consensus_queue.cc:1035] T 543054d6241649d59bb40156217a73bc P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:59:14.440894 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:14.441121 31693 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:14.441840 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:14.441907 1428 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:14.442488 1428 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.442824 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:14.442955 1428 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.441907 1252 consensus_queue.cc:1046] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20250902 01:59:14.431318 1408 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:14.444147 1252 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:14.444659 1252 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.445037 1428 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:14.446506 1252 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:14.447083 1252 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:14.452852 30919 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:14.443411 30917 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:14.435174 1317 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.460933 1408 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.464139 1402 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.405727 1407 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.427724 1417 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:14.405396 1147 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:14.476167 31693 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:14.471282 1459 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.420799 1329 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.424042 1141 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:14.517846 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:14.592976 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 116882us
I20250902 01:59:14.447448 1252 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:14.466504 1255 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.422263 1425 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:14.593935 31693 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:14.479202 1285 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:14.425130 1244 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:14.672541 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 271913us
W20250902 01:59:14.681399 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 87556.8us
I20250902 01:59:14.691113 1483 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:14.479084 1449 consensus_queue.cc:1046] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.206s
I20250902 01:59:14.586614 1451 raft_consensus.cc:2747] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:14.703183 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:14.425400 867 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:14.721390 30924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.017s sys 0.010s
W20250902 01:59:14.681429 1446 log.cc:927] Time spent T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
W20250902 01:59:14.709172 30925 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.004s sys 0.017s
I20250902 01:59:14.623493 1252 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:14.642547 1147 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:14.590111 1417 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:14.721725 30924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s user 0.017s sys 0.010s
I20250902 01:59:14.746789 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1732) took 1971 ms. Trace:
I20250902 01:59:14.747246 30924 rpcz_store.cc:276] 0902 01:59:12.775259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.219419 (+444160us) service_pool.cc:225] Handling call
0902 01:59:13.219752 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:13.219971 (+ 219us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:13.912691 (+692720us) log.cc:844] Serialized 14 byte log entry
0902 01:59:14.746310 (+833619us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:14.746398 (+ 88us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:14.746461 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.746553 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:14.746723 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42220,"prepare.queue_time_us":275704,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":1466444,"threads_started":2}
I20250902 01:59:14.697547 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:14.640756 1426 raft_consensus.cc:695] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:14.657542 1328 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:14.691182 1141 consensus_queue.cc:260] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:14.730827 31071 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.042s sys 0.060s
W20250902 01:59:14.727403 30925 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s user 0.004s sys 0.017s
W20250902 01:59:14.749752 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 52317us
I20250902 01:59:14.465677 1427 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:14.751693 31071 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s user 0.042s sys 0.060s
W20250902 01:59:14.750298 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 171124us
I20250902 01:59:14.752141 31038 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:14.691908 1425 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:14.504061 1321 consensus_queue.cc:260] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:14.601831 1329 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.681241 1413 consensus_queue.cc:1035] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.415s
I20250902 01:59:14.656512 1255 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.741053 1147 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:14.807982 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 449562us
I20250902 01:59:14.740154 1325 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:14.750888 1328 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:14.751415 1417 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:14.751113 1272 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:14.726012 1450 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250902 01:59:14.902261 30965 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:14.902163 1445 log.cc:927] Time spent T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.490s user 0.000s sys 0.001s
I20250902 01:59:14.640790 1407 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.903307 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 660) took 1093 ms. Trace:
I20250902 01:59:14.623528 1459 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:14.793838 1464 consensus_queue.cc:1046] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.463s
I20250902 01:59:14.762501 1375 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:14.762094 1451 consensus_queue.cc:1035] T 59bc91ec50764c7bb821a4288bea8d16 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.098s
I20250902 01:59:14.720099 1483 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:14.818032 1147 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:14.932269 31729 rpcz_store.cc:276] 0902 01:59:13.809933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.810026 (+ 93us) service_pool.cc:225] Handling call
0902 01:59:14.903055 (+1093029us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b540059a954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:14.903226 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492037376}
W20250902 01:59:14.862010 1456 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.501s user 0.001s sys 0.000s
W20250902 01:59:14.857928 1477 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.233s user 0.001s sys 0.000s
I20250902 01:59:14.868314 1321 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
I20250902 01:59:14.869747 1328 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:14.957002 31722 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.014s sys 0.051s
I20250902 01:59:14.825953 1366 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:14.903406 1455 log.cc:927] Time spent T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
I20250902 01:59:15.159250 522 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
W20250902 01:59:14.907127 1298 consensus_meta.cc:287] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.034s user 0.007s sys 0.000s
I20250902 01:59:15.159674 522 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:15.159987 1298 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:15.160259 1084 consensus_meta.cc:287] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.755s user 0.013s sys 0.002s
I20250902 01:59:15.160424 1298 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.160653 1084 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.161806 1381 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:14.894624 1141 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:14.872799 1292 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:14.902189 1386 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:14.884125 1325 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:14.908289 1406 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.033140 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:15.040235 1459 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:15.040452 1384 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.041427 1255 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.033434 1375 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election won for term 5
W20250902 01:59:15.044111 1388 consensus_meta.cc:287] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.173s user 0.006s sys 0.003s
I20250902 01:59:15.045459 1407 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:14.985224 1359 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:14.934274 1411 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:14.293537 31723 rpcz_store.cc:276] 0902 01:59:13.003555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.012775 (+ 9220us) service_pool.cc:225] Handling call
0902 01:59:13.013089 (+ 314us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:13.303997 (+290908us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.304045 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:13.304778 (+ 733us) log.cc:844] Serialized 52 byte log entry
0902 01:59:13.365213 (+ 60435us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:14.096605 (+731392us) log.cc:844] Serialized 14 byte log entry
0902 01:59:14.145214 (+ 48609us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.153546 (+ 8332us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:14.159986 (+ 6440us) raft_consensus.cc:1683] finished
0902 01:59:14.165888 (+ 5902us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:14.187561 (+ 21673us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39871,"prepare.queue_time_us":914561,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":321892,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":28817,"thread_start_us":990002,"threads_started":3,"wal-append.queue_time_us":161776}
I20250902 01:59:15.401814 1359 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:15.402395 1359 raft_consensus.cc:695] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:14.097852 30923 rpcz_store.cc:276] 0902 01:59:12.774925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.217146 (+442221us) service_pool.cc:225] Handling call
0902 01:59:13.217446 (+ 300us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:13.217692 (+ 246us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:13.805360 (+587668us) log.cc:844] Serialized 14 byte log entry
0902 01:59:13.988466 (+183106us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:13.988525 (+ 59us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:13.988577 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:13.988650 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:13.988774 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":243367,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":108458,"thread_start_us":725826,"threads_started":2}
I20250902 01:59:15.363098 1388 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250902 01:59:15.403389 1359 consensus_queue.cc:237] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:15.404632 30995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1909) took 1226 ms (client timeout 1576 ms). Trace:
I20250902 01:59:15.194746 1366 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:15.404986 30995 rpcz_store.cc:269] 0902 01:59:14.178427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.190661 (+ 12234us) service_pool.cc:225] Handling call
0902 01:59:15.404486 (+1213825us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54002bdc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.404585 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342948224}
I20250902 01:59:15.066563 1317 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:15.375317 1298 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.408528 1512 raft_consensus.cc:1062] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:59:15.336463 1406 raft_consensus.cc:3053] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:15.139773 1461 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:15.140599 1460 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.332s user 0.000s sys 0.000s
W20250902 01:59:15.144837 1478 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20250902 01:59:14.796044 1425 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:15.012009 1467 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:14.983445 1448 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.099859 1408 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.147531 1402 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:15.148101 1480 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.239s user 0.000s sys 0.000s
I20250902 01:59:15.087011 1472 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.148808 1329 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:15.158147 31722 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.989s user 0.015s sys 0.052s
I20250902 01:59:14.826712 1466 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.548238 1466 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.550158 1466 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:14.966742 31728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.013s sys 0.005s
W20250902 01:59:15.551501 31728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.423s user 0.013s sys 0.005s
I20250902 01:59:15.551854 1472 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.552461 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 662) took 1741 ms. Trace:
I20250902 01:59:15.053418 1426 consensus_queue.cc:237] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:15.179281 31070 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 595) took 2243 ms (client timeout 1829 ms). Trace:
I20250902 01:59:15.553979 1472 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:15.553809 31070 rpcz_store.cc:269] 0902 01:59:12.936012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:12.936178 (+ 166us) service_pool.cc:225] Handling call
0902 01:59:15.179084 (+2242906us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54001e7dd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.179198 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1321557888}
I20250902 01:59:15.524575 1401 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:15.555105 1401 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.182879 1084 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.556423 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 661) took 1745 ms. Trace:
I20250902 01:59:15.557688 1401 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:15.556835 31722 rpcz_store.cc:276] 0902 01:59:13.810832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.825542 (+ 14710us) service_pool.cc:225] Handling call
0902 01:59:13.825833 (+ 291us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.831661 (+ 5828us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.831694 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.874293 (+ 42599us) log.cc:844] Serialized 26 byte log entry
0902 01:59:13.975598 (+101305us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:14.169090 (+193492us) log.cc:844] Serialized 14 byte log entry
0902 01:59:15.524348 (+1355258us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.524414 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.556184 (+ 31770us) raft_consensus.cc:1683] finished
0902 01:59:15.556203 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.556369 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":373678,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":5894,"raft.queue_time_us":264186,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2622,"thread_start_us":1589409,"threads_started":3,"wal-append.queue_time_us":1661115}
W20250902 01:59:15.557988 1420 log.cc:927] Time spent T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 1.658s user 0.011s sys 0.016s
I20250902 01:59:15.524744 1448 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.183517 1414 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:15.560092 1448 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.560912 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 948) took 2622 ms. Trace:
I20250902 01:59:15.183997 1244 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:15.160458 522 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:15.561262 31724 rpcz_store.cc:276] 0902 01:59:12.938233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:12.943306 (+ 5073us) service_pool.cc:225] Handling call
0902 01:59:13.019592 (+ 76286us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.093938 (+ 74346us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.098012 (+ 4074us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.260654 (+162642us) log.cc:844] Serialized 26 byte log entry
0902 01:59:13.427517 (+166863us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:13.450253 (+ 22736us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:13.456648 (+ 6395us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.557358 (+2100710us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:15.560545 (+ 3187us) raft_consensus.cc:1683] finished
0902 01:59:15.560578 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.560842 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13654,"raft.queue_time_us":15507,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":37770,"spinlock_wait_cycles":621474176,"thread_start_us":63132,"threads_started":1,"wal-append.queue_time_us":535013}
W20250902 01:59:15.184753 1364 consensus_meta.cc:287] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.860s user 0.007s sys 0.006s
I20250902 01:59:15.185094 1483 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:15.185945 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:15.563722 30924 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:15.563313 1364 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.186403 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 653) took 1382 ms. Trace:
I20250902 01:59:15.564311 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:15.564493 31727 rpcz_store.cc:276] 0902 01:59:13.804937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.805229 (+ 292us) service_pool.cc:225] Handling call
0902 01:59:13.829129 (+ 23900us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:13.874543 (+ 45414us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.186146 (+1311603us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b5400599cd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:15.186340 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89151232}
I20250902 01:59:15.565761 1364 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:15.187126 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.120s user 0.037s sys 0.089s
I20250902 01:59:15.566118 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53742 (request call id 1020) took 1384 ms. Trace:
I20250902 01:59:15.162262 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:15.566555 30996 rpcz_store.cc:276] 0902 01:59:14.181584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.429909 (+248325us) service_pool.cc:225] Handling call
0902 01:59:15.565608 (+1135699us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540004f1d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b4cd0 00007fc602138eea 00007fc60709160e 00007fc60072ecc7 00007fc60072ec52 00007fc60072eb81 00007fc60072eafc 00007fc60072ddb2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.566017 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401146240}
I20250902 01:59:15.567516 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1908) took 1389 ms. Trace:
I20250902 01:59:15.191006 31072 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:15.182358 1479 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.184242 1427 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.353381 1325 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.229614 1386 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:15.255435 1292 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.295648 1475 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:15.337734 30965 leader_election.cc:336] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.901s (SENT)
I20250902 01:59:15.338665 31729 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:15.364755 30993 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.267s user 0.012s sys 0.024s
I20250902 01:59:15.334926 1379 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:15.392693 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.010s sys 0.029s
W20250902 01:59:15.393288 30992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.009s sys 0.030s
W20250902 01:59:15.394338 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.040s sys 0.054s
W20250902 01:59:15.398062 31725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.012s sys 0.031s
I20250902 01:59:15.017900 30926 raft_consensus.cc:3053] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
W20250902 01:59:15.132201 1444 log.cc:927] Time spent T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.427s user 0.001s sys 0.001s
W20250902 01:59:15.432094 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.007s sys 0.044s
W20250902 01:59:15.461804 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.082s user 0.010s sys 0.006s
I20250902 01:59:15.406733 1366 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:15.472469 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.066s user 0.002s sys 0.018s
I20250902 01:59:15.355816 1255 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:15.431965 1270 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.008s sys 0.023s
W20250902 01:59:15.432363 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.005s sys 0.025s
I20250902 01:59:15.426823 1406 raft_consensus.cc:738] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:15.421224 1375 raft_consensus.cc:695] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:15.522992 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.097s user 0.014s sys 0.022s
I20250902 01:59:15.410691 1512 consensus_queue.cc:237] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:15.535379 1252 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.772s user 0.006s sys 0.003s
I20250902 01:59:15.523028 1328 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.524160 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 550) took 1642 ms. Trace:
I20250902 01:59:15.537506 1461 raft_consensus.cc:695] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:15.538511 1495 log.cc:927] Time spent T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20250902 01:59:15.546155 1147 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:15.547240 1458 log.cc:927] Time spent T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.956s user 0.008s sys 0.000s
W20250902 01:59:15.547864 1417 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.679s user 0.005s sys 0.005s
I20250902 01:59:15.553180 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 547) took 1677 ms. Trace:
I20250902 01:59:15.182449 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 460) took 1270 ms. Trace:
W20250902 01:59:15.566406 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.531s user 0.037s sys 0.090s
I20250902 01:59:15.567832 31001 rpcz_store.cc:276] 0902 01:59:14.178097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.208367 (+ 30270us) service_pool.cc:225] Handling call
0902 01:59:15.567360 (+1358993us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b5400071054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.567468 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2191713536}
I20250902 01:59:15.194991 1381 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:15.566151 31693 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:15.695130 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 129083us
I20250902 01:59:15.695720 1364 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:15.696208 1328 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:15.697032 1270 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.921s user 0.008s sys 0.024s
I20250902 01:59:15.696348 30920 rpcz_store.cc:276] 0902 01:59:13.875518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.875697 (+ 179us) service_pool.cc:225] Handling call
0902 01:59:13.899154 (+ 23457us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:14.032091 (+132937us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:14.032132 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.075798 (+ 43666us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.284114 (+208316us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:14.284362 (+ 248us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.284782 (+ 420us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.552999 (+1268217us) raft_consensus.cc:1683] finished
0902 01:59:15.553018 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.553131 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18092,"raft.queue_time_us":123969,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":44971,"thread_start_us":248160,"threads_started":2,"wal-append.queue_time_us":1446861}
I20250902 01:59:15.697974 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 603) took 2363 ms. Trace:
I20250902 01:59:15.698280 31074 rpcz_store.cc:276] 0902 01:59:13.334910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.335021 (+ 111us) service_pool.cc:225] Handling call
0902 01:59:15.697828 (+2362807us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b5400201054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.697928 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68487552}
I20250902 01:59:15.698614 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1905) took 1541 ms. Trace:
I20250902 01:59:15.699267 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 659) took 1889 ms. Trace:
W20250902 01:59:15.699659 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.293s user 0.007s sys 0.022s
W20250902 01:59:15.700018 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.29364e+06us
I20250902 01:59:15.700053 1024 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:15.700402 1417 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.700660 1024 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.696112 1364 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:15.702278 1024 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.702446 1459 raft_consensus.cc:1025] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: attempt to promote peer 01f6731ce5044a6a948a5ae04da80e6f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250902 01:59:15.702452 1364 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:15.703243 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 961) took 1723 ms. Trace:
I20250902 01:59:15.703564 31726 rpcz_store.cc:276] 0902 01:59:13.979578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.187199 (+207621us) service_pool.cc:225] Handling call
0902 01:59:15.703074 (+1515875us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400312754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.703196 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":335736704}
W20250902 01:59:15.704361 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.278s user 0.015s sys 0.023s
I20250902 01:59:15.699667 31730 rpcz_store.cc:276] 0902 01:59:13.809611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.809704 (+ 93us) service_pool.cc:225] Handling call
0902 01:59:13.872516 (+ 62812us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:13.874897 (+ 2381us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.699120 (+1824223us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b54005901d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:15.699222 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93276928}
W20250902 01:59:15.705960 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.327s user 0.010s sys 0.009s
I20250902 01:59:15.698920 31000 rpcz_store.cc:276] 0902 01:59:14.157577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.228756 (+ 71179us) service_pool.cc:225] Handling call
0902 01:59:14.229479 (+ 723us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:14.361004 (+131525us) raft_consensus.cc:1532] Early marking committed up to index 4
0902 01:59:14.386047 (+ 25043us) log.cc:844] Serialized 14 byte log entry
0902 01:59:14.604746 (+218699us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:14.604793 (+ 47us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:14.604835 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.698408 (+1093573us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540004f1d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:15.698445 (+ 37us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.698572 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25180,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":684034,"spinlock_wait_cycles":309325440,"thread_start_us":218351,"threads_started":2,"wal-append.queue_time_us":393332}
I20250902 01:59:15.696113 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1703) took 1296 ms. Trace:
W20250902 01:59:15.706825 31039 connection.cc:232] RPC call timeout handler was delayed by -1.2363041720000183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
W20250902 01:59:15.708716 31039 connection.cc:232] RPC call timeout handler was delayed by -1.0714528510001173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20250902 01:59:15.708402 31074 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:38110
W20250902 01:59:15.708686 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.942s user 0.006s sys 0.025s
W20250902 01:59:15.709339 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:15.710407 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.052s user 0.008s sys 0.045s
I20250902 01:59:15.707770 31072 rpcz_store.cc:276] 0902 01:59:14.399716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.399835 (+ 119us) service_pool.cc:225] Handling call
0902 01:59:14.420568 (+ 20733us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:15.568024 (+1147456us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.695922 (+127898us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b540024b3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:15.696061 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1705909504}
I20250902 01:59:15.707816 30918 rpcz_store.cc:276] 0902 01:59:13.881351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.881472 (+ 121us) service_pool.cc:225] Handling call
0902 01:59:13.900703 (+ 19231us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:14.167645 (+266942us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:14.167685 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.168210 (+ 525us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.325318 (+157108us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:14.325430 (+ 112us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.325602 (+ 172us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.523956 (+1198354us) raft_consensus.cc:1683] finished
0902 01:59:15.523976 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.524111 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17032,"raft.queue_time_us":176828,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":9955,"thread_start_us":406355,"threads_started":2,"wal-append.queue_time_us":1048965}
I20250902 01:59:15.711308 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:15.707547 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 1.67247e+06us
I20250902 01:59:15.710250 30999 rpcz_store.cc:276] 0902 01:59:13.911907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.913043 (+ 1136us) service_pool.cc:225] Handling call
0902 01:59:13.913372 (+ 329us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.916948 (+ 3576us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.916989 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.989718 (+ 72729us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.185674 (+195956us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:14.189092 (+ 3418us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.189199 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:14.901738 (+712539us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b48001503d8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
0902 01:59:15.182145 (+280407us) raft_consensus.cc:1683] finished
0902 01:59:15.182168 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.182392 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3064,"raft.queue_time_us":3186,"raft.run_cpu_time_us":16786,"raft.run_wall_time_us":973402,"spinlock_wait_cycles":113197440,"thread_start_us":168014,"threads_started":1,"wal-append.queue_time_us":207473}
W20250902 01:59:15.709470 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 957418us
W20250902 01:59:15.714591 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:15.715667 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:15.717046 31725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.978s user 0.014s sys 0.033s
W20250902 01:59:15.717605 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.966s user 0.010s sys 0.035s
W20250902 01:59:15.718204 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:15.718254 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 964) took 1023 ms. Trace:
W20250902 01:59:15.718673 30992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.012s sys 0.033s
I20250902 01:59:15.719400 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1911) took 1471 ms. Trace:
W20250902 01:59:15.719951 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.904s user 0.042s sys 0.056s
I20250902 01:59:15.718704 31725 rpcz_store.cc:276] 0902 01:59:14.694986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.710151 (+ 15165us) service_pool.cc:225] Handling call
0902 01:59:14.710488 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:14.710781 (+ 293us) raft_consensus.cc:1532] Early marking committed up to index 9
0902 01:59:14.739107 (+ 28326us) log.cc:844] Serialized 14 byte log entry
0902 01:59:15.717742 (+978635us) log.cc:844] Serialized 14 byte log entry
0902 01:59:15.717945 (+ 203us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:15.717991 (+ 46us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:15.718036 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.718100 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.718204 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39417,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"thread_start_us":978028,"threads_started":1,"wal-append.queue_time_us":823886}
I20250902 01:59:15.720618 30965 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:15.719978 30992 rpcz_store.cc:276] 0902 01:59:14.247905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.354396 (+106491us) service_pool.cc:225] Handling call
0902 01:59:14.386188 (+ 31792us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:14.696064 (+309876us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:14.696094 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.696553 (+ 459us) log.cc:844] Serialized 26 byte log entry
0902 01:59:15.719135 (+1022582us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:15.719172 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.719224 (+ 52us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.719261 (+ 37us) raft_consensus.cc:1683] finished
0902 01:59:15.719271 (+ 10us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.719358 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11751,"raft.queue_time_us":701217,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":47422,"thread_start_us":1250518,"threads_started":2,"wal-append.queue_time_us":978065}
I20250902 01:59:15.721248 30926 raft_consensus.cc:738] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:15.721719 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 968) took 1017 ms. Trace:
I20250902 01:59:15.615303 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:15.722359 30923 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:15.722592 30993 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.630s user 0.012s sys 0.032s
W20250902 01:59:15.618835 31041 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.000s sys 0.024s
I20250902 01:59:15.640169 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 552) took 1629 ms. Trace:
I20250902 01:59:15.714483 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:15.722127 31729 rpcz_store.cc:276] 0902 01:59:14.704545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.032910 (+328365us) service_pool.cc:225] Handling call
0902 01:59:15.721666 (+688756us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:15.723222 31041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.000s sys 0.035s
I20250902 01:59:15.570931 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 549) took 1689 ms. Trace:
I20250902 01:59:15.725412 30925 rpcz_store.cc:276] 0902 01:59:13.881034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.881140 (+ 106us) service_pool.cc:225] Handling call
0902 01:59:13.881500 (+ 360us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:14.123956 (+242456us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:14.123997 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.124457 (+ 460us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.751402 (+626945us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:14.751453 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.751524 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.570654 (+819130us) raft_consensus.cc:1683] finished
0902 01:59:15.570701 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.570862 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":481,"raft.queue_time_us":47146,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":60493,"thread_start_us":803262,"threads_started":2,"wal-append.queue_time_us":1376652}
W20250902 01:59:15.603340 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.286s user 0.008s sys 0.027s
I20250902 01:59:15.552856 31728 rpcz_store.cc:276] 0902 01:59:13.811054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.827444 (+ 16390us) service_pool.cc:225] Handling call
0902 01:59:13.832512 (+ 5068us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.832895 (+ 383us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.832931 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.868075 (+ 35144us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.095191 (+227116us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:15.551896 (+1456705us) log.cc:844] Serialized 14 byte log entry
0902 01:59:15.552169 (+ 273us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.552253 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.552293 (+ 40us) raft_consensus.cc:1683] finished
0902 01:59:15.552306 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.552415 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":285815,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":40558,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":8707,"thread_start_us":1592441,"threads_started":2,"wal-append.queue_time_us":597380}
W20250902 01:59:15.728578 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.412s user 0.008s sys 0.033s
I20250902 01:59:15.723424 30922 rpcz_store.cc:276] 0902 01:59:14.010459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:14.010616 (+ 157us) service_pool.cc:225] Handling call
0902 01:59:14.011061 (+ 445us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:14.023517 (+ 12456us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:14.023555 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.068275 (+ 44720us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.237321 (+169046us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:14.237372 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.237460 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.148520 (+911060us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b4800080058. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
0902 01:59:15.567233 (+418713us) raft_consensus.cc:1683] finished
0902 01:59:15.567254 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.640107 (+ 72853us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12061,"raft.queue_time_us":93107,"raft.run_cpu_time_us":10082,"raft.run_wall_time_us":1051332,"spinlock_wait_cycles":48935936,"thread_start_us":94471,"threads_started":1,"wal-append.queue_time_us":611867}
I20250902 01:59:15.722111 30926 consensus_queue.cc:260] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:15.690027 1423 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.666s user 0.002s sys 0.000s
I20250902 01:59:15.732132 30889 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:15.737027 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 399322us
W20250902 01:59:15.619338 1439 log.cc:927] Time spent T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250902 01:59:15.734766 30926 raft_consensus.cc:3058] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:15.461133 1405 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.730053 1506 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.746910 1508 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:15.767266 1471 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:15.730612 1475 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.702280 1417 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:15.714144 1323 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.752519 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:15.773010 30918 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:15.773211 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.416s user 0.038s sys 0.034s
I20250902 01:59:15.728641 1386 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:15.755173 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:15.714365 1400 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:15.752390 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:15.730481 1366 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.790468 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:15.751765 1376 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.796623 1505 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.796438 1376 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.797128 1505 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.798396 1376 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.799208 30996 raft_consensus.cc:1273] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:15.736622 1479 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.799883 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:15.800595 31729 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:15.800895 31070 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:15.730473 1379 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.766732 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1899) took 2478 ms. Trace:
I20250902 01:59:15.767304 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 462) took 1836 ms. Trace:
I20250902 01:59:15.751300 30967 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:15.802142 1425 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:15.802484 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 51288.7us
W20250902 01:59:15.804351 1441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.026s sys 0.079s
I20250902 01:59:15.769920 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:15.765610 1506 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:15.740224 1359 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20250902 01:59:15.730842 1292 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.517796 1467 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.764408 1472 consensus_queue.cc:1035] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.087s
I20250902 01:59:15.779458 30889 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:15.782933 30926 raft_consensus.cc:1273] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20250902 01:59:15.758224 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:15.788583 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:15.790086 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:15.725065 1252 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.568524 1413 raft_consensus.cc:1062] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
I20250902 01:59:15.795210 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1696) took 1977 ms. Trace:
I20250902 01:59:15.795512 1513 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:15.795954 1469 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:15.799762 1400 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.802649 1379 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:15.804246 1438 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:15.795149 1482 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:15.735352 1255 consensus_queue.cc:1035] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20250902 01:59:15.775950 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:15.776144 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:15.773674 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.416s user 0.038s sys 0.034s
I20250902 01:59:15.806373 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:15.909564 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.231639 31729 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:15.921937 31075 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 2 FOLLOWER]: Advancing to term 3
W20250902 01:59:15.804762 1441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.026s sys 0.079s
I20250902 01:59:15.767884 1471 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:15.842257 30923 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20250902 01:59:16.007427 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:15.896209 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 602) took 2561 ms. Trace:
I20250902 01:59:15.801784 30994 rpcz_store.cc:276] 0902 01:59:13.930606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.930757 (+ 151us) service_pool.cc:225] Handling call
0902 01:59:13.931154 (+ 397us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.931616 (+ 462us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.931651 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.932233 (+ 582us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.215312 (+283079us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:14.215349 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.215410 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.767092 (+1551682us) raft_consensus.cc:1683] finished
0902 01:59:15.767116 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.767250 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":49649,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":5582,"thread_start_us":189440,"threads_started":1,"wal-append.queue_time_us":1775933}
W20250902 01:59:16.150236 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:16.180482 31071 rpcz_store.cc:276] 0902 01:59:13.817829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.826129 (+ 8300us) service_pool.cc:225] Handling call
0902 01:59:13.830276 (+ 4147us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.842318 (+ 12042us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.842358 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.881922 (+ 39564us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.765999 (+884077us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:14.766037 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.766101 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.561079 (+794978us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b48002715d8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
0902 01:59:15.794924 (+233845us) raft_consensus.cc:1683] finished
0902 01:59:15.794951 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.795156 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11625,"raft.queue_time_us":12221,"raft.run_cpu_time_us":15585,"raft.run_wall_time_us":1692881,"spinlock_wait_cycles":8088448,"thread_start_us":833270,"threads_started":1,"wal-append.queue_time_us":1896312}
I20250902 01:59:16.155032 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 548) took 2274 ms. Trace:
I20250902 01:59:16.139207 30918 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
W20250902 01:59:16.027046 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 51455.3us
I20250902 01:59:15.751956 1499 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:16.203754 31725 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:15.875751 1443 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:16.174093 31728 raft_consensus.cc:2408] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:15.807679 31001 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:15.801685 30998 rpcz_store.cc:276] 0902 01:59:13.288639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.288855 (+ 216us) service_pool.cc:225] Handling call
0902 01:59:13.322445 (+ 33590us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.322726 (+ 281us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.322752 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.323150 (+ 398us) log.cc:844] Serialized 26 byte log entry
0902 01:59:13.579711 (+256561us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:13.579763 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:13.579854 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.689447 (+2109593us) spinlock_profiling.cc:243] Waited 93.5 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:15.766433 (+ 76986us) raft_consensus.cc:1683] finished
0902 01:59:15.766453 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.766659 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61444,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6632,"spinlock_wait_cycles":215064320,"thread_start_us":225033,"threads_started":1,"wal-append.queue_time_us":539979}
I20250902 01:59:16.288156 1443 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.908159 1381 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.904968 1475 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.208770 1024 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:16.022876 1461 consensus_queue.cc:237] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.290189 31000 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:16.206008 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.290868 30996 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:16.291815 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:16.292352 30996 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:16.280092 1482 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.294373 1482 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:15.890630 1375 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:15.771319 1402 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:16.296259 1402 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:16.211478 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 432097us
I20250902 01:59:16.233598 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.234244 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.298182 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.298249 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.298907 31067 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:16.299240 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.299717 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.236853 31729 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250902 01:59:16.300226 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.300886 31073 raft_consensus.cc:2366] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 8. Current term is 9.
W20250902 01:59:16.301509 1508 consensus_meta.cc:287] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.534s user 0.003s sys 0.005s
I20250902 01:59:16.300248 31067 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:16.302309 1508 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:16.301527 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.174826 1469 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:16.238761 1398 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250902 01:59:16.123922 1448 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:16.261204 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.318830 31075 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:59:16.242690 31069 rpcz_store.cc:276] 0902 01:59:13.334605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.334723 (+ 118us) service_pool.cc:225] Handling call
0902 01:59:13.335017 (+ 294us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.335205 (+ 188us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.335231 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.336207 (+ 976us) log.cc:844] Serialized 26 byte log entry
0902 01:59:13.834005 (+497798us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:13.834052 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:13.834117 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.895987 (+2061870us) raft_consensus.cc:1683] finished
0902 01:59:15.896013 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:15.896154 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":487032,"threads_started":1,"wal-append.queue_time_us":2006534}
I20250902 01:59:16.275315 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.282400 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.320886 30994 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:16.321766 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.322702 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:15.855443 1505 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:16.333211 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 71478.6us
I20250902 01:59:16.323391 30994 raft_consensus.cc:2391] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:16.336268 30999 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:16.246821 30921 rpcz_store.cc:276] 0902 01:59:13.880679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.880803 (+ 124us) service_pool.cc:225] Handling call
0902 01:59:13.895503 (+ 14700us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.917499 (+ 21996us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:14.011283 (+ 93784us) log.cc:844] Serialized 14 byte log entry
0902 01:59:14.370512 (+359229us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:14.388924 (+ 18412us) log.cc:844] Serialized 26 byte log entry
0902 01:59:14.389079 (+ 155us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:14.389119 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:14.389202 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:15.180716 (+791514us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b180044aa60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff19c786 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61
0902 01:59:15.181188 (+ 472us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b180044aa60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:15.546500 (+365312us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:16.154764 (+608264us) raft_consensus.cc:1683] finished
0902 01:59:16.154793 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:16.154966 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":174373,"prepare.queue_time_us":93614,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":566187,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":53386,"spinlock_wait_cycles":66567040,"thread_start_us":108882,"threads_started":1,"wal-append.queue_time_us":319737}
I20250902 01:59:15.791282 1272 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250902 01:59:16.185696 1412 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.023s sys 0.058s
I20250902 01:59:16.259256 1471 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:16.289181 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.231840 30992 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:16.303236 1376 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:16.347180 30998 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:16.302523 30996 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:16.174052 1438 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:16.304646 31073 raft_consensus.cc:2408] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:16.305140 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:16.258383 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:16.311403 1469 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:15.905994 1147 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:16.211143 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 538) took 3680 ms. Trace:
I20250902 01:59:16.260514 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.303699 1421 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:16.309317 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:16.229650 1413 raft_consensus.cc:1066] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Unable to promote non-voter 01f6731ce5044a6a948a5ae04da80e6f: Illegal state: Leader has not yet committed an operation in its own term
I20250902 01:59:16.173620 1513 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:16.258410 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.261965 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:16.395787 31730 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:15.783278 1323 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.398403 1323 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.399603 31068 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:16.402356 31692 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:16.331568 31071 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:15.793262 1405 consensus_queue.cc:1035] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.486s
I20250902 01:59:16.341949 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:15.896077 1464 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.424s
I20250902 01:59:16.405426 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:15.774693 1406 consensus_queue.cc:260] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.279579 1400 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:16.349279 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:16.357924 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.411916 30992 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
W20250902 01:59:16.359212 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 54152us
I20250902 01:59:16.360361 30926 rpcz_store.cc:276] 0902 01:59:12.531207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.193759 (+662552us) service_pool.cc:225] Handling call
0902 01:59:13.194109 (+ 350us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:15.017708 (+1823599us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54002524a0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020bf550 00007fc602132170 00007fc6021331e1 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c
0902 01:59:16.199701 (+1181993us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:16.211092 (+ 11391us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276386688}
I20250902 01:59:16.369055 30918 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:16.360071 30997 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:16.360785 30923 raft_consensus.cc:2408] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:16.350368 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:16.322957 1503 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:16.334529 1426 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.016s sys 0.028s
I20250902 01:59:16.347184 1272 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:15.850679 1366 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.351711 1438 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:16.346977 1325 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.302676 1386 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.350788 1376 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.426158 30998 raft_consensus.cc:3058] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:16.405335 31730 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:16.351328 1381 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.351020 1292 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.353696 1506 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:16.347757 1412 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.797s user 0.026s sys 0.058s
I20250902 01:59:16.414812 31692 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:16.433491 1436 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:16.438812 1364 consensus_meta.cc:287] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.736s user 0.007s sys 0.004s
I20250902 01:59:16.356626 1402 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.354502 1252 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.359697 1482 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:16.407382 31038 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:16.434756 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:16.478644 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 71343.2us
I20250902 01:59:16.479326 31038 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:16.485670 30918 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
W20250902 01:59:16.428825 1426 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s user 0.017s sys 0.028s
I20250902 01:59:16.412393 1359 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:16.448873 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.373364 1448 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:16.465554 1475 consensus_queue.cc:1035] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20250902 01:59:16.432067 1272 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:16.407733 1421 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:16.429697 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:16.430248 30889 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:16.435773 1508 consensus_queue.cc:1162] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "01f6731ce5044a6a948a5ae04da80e6f" responder_term: 6 status { last_received { term: 4 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20250902 01:59:16.505182 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 75005.2us
I20250902 01:59:16.494781 1413 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:16.481611 1402 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:16.489027 31695 connection.cc:232] RPC call timeout handler was delayed by -1.8671182439999257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20250902 01:59:16.489602 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 74881.6us
I20250902 01:59:16.514784 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.476426 1436 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.475179 1472 raft_consensus.cc:2762] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250902 01:59:16.495863 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:16.426836 1424 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.006s sys 0.035s
I20250902 01:59:16.482113 1466 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:16.406255 1513 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:16.476868 1506 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:16.502210 31072 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.013s sys 0.029s
W20250902 01:59:16.551805 31072 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.801s user 0.013s sys 0.030s
I20250902 01:59:16.487149 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:16.552798 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 65749.6us
I20250902 01:59:16.545450 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:16.553952 30967 leader_election.cc:400] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 8. Current term is 9.
I20250902 01:59:16.554314 30967 leader_election.cc:403] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20250902 01:59:16.507614 30926 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:59:16.508977 30923 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:16.555626 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:16.556382 30923 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:16.557060 31726 raft_consensus.cc:2391] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:16.555454 31039 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.009s sys 0.028s
I20250902 01:59:16.510672 1421 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:16.475621 1364 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.515497 30924 raft_consensus.cc:2374] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8. Re-sending same reply.
I20250902 01:59:16.523888 1467 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.516575 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.517192 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.524451 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.525991 30998 raft_consensus.cc:2466] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:16.567046 31728 raft_consensus.cc:2391] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:16.567653 30998 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250902 01:59:16.533143 1468 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:16.438203 1503 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.529918 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.530133 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.530339 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.531451 31075 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250902 01:59:16.581856 31069 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
I20250902 01:59:16.582370 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.582844 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.499850 1359 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:16.535822 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.585363 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.585551 31070 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:16.536504 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.537189 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.586433 31068 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:16.544760 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:16.528767 1426 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20250902 01:59:16.593925 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.563635 1506 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:16.594897 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.595461 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:16.550239 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.596053 31068 raft_consensus.cc:3058] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:16.596316 30920 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:16.596647 1506 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.597110 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.597298 1506 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 9
I20250902 01:59:16.597802 30920 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:16.598486 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.599001 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.587239 1468 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.599254 30920 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:16.551040 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:16.559161 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:16.557960 31039 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.701s user 0.010s sys 0.028s
I20250902 01:59:16.559211 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.524770 1405 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:16.481869 1482 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:16.557958 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.512578 1413 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:16.563097 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.622222 30924 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:16.532967 1402 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.524989 1430 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:16.563824 30921 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:16.565528 30967 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:16.631639 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 66199.7us
I20250902 01:59:16.568976 30964 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:16.633345 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 64444.4us
I20250902 01:59:16.526000 31695 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:16.635272 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 109350us
I20250902 01:59:16.569192 31692 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:16.527968 1443 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.583611 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.584340 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:16.542532 1424 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.852s user 0.006s sys 0.035s
I20250902 01:59:16.594959 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.556732 1466 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:16.599478 31730 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:16.549176 31038 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:16.664342 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 115269us
I20250902 01:59:16.664973 31038 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:16.607515 31068 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:16.600895 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.617023 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.666319 31728 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:16.618861 30923 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:16.619308 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.515865 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:16.586699 31071 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:16.599905 30919 raft_consensus.cc:2391] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 3: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:16.600747 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:16.636595 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 67486.6us
I20250902 01:59:16.637982 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.600342 1292 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:16.600131 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 743047us
I20250902 01:59:16.639807 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:16.639915 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.599874 1359 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:16.640229 31066 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.014s sys 0.032s
W20250902 01:59:16.651942 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:16.662204 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.666688 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:16.995532 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.018s sys 0.029s
W20250902 01:59:16.932601 30922 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.013s sys 0.029s
W20250902 01:59:17.086710 30922 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.771s user 0.018s sys 0.037s
W20250902 01:59:16.951279 30891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.007s sys 0.037s
I20250902 01:59:16.681835 30992 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
W20250902 01:59:17.087541 30891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.010s sys 0.044s
I20250902 01:59:17.087960 31075 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:17.088390 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:16.974604 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.013s sys 0.054s
W20250902 01:59:17.089429 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.013s sys 0.064s
I20250902 01:59:17.089815 31726 raft_consensus.cc:1273] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250902 01:59:16.667912 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:17.091548 30920 raft_consensus.cc:2374] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
W20250902 01:59:17.089336 31039 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.775s (SENT)
W20250902 01:59:17.086181 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.021s sys 0.033s
W20250902 01:59:17.092695 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 782422us
W20250902 01:59:17.092871 31039 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.864s (SENT)
I20250902 01:59:16.797137 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:17.093763 31039 leader_election.cc:336] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.897s (SENT)
I20250902 01:59:16.628397 1413 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:17.093099 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:17.101639 31730 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
W20250902 01:59:16.740578 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:17.085523 31066 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.961s user 0.042s sys 0.038s
W20250902 01:59:17.094662 31039 leader_election.cc:336] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.985s (SENT)
W20250902 01:59:16.940665 30995 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.005s sys 0.037s
I20250902 01:59:17.102903 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:17.103127 30995 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.799s user 0.005s sys 0.049s
W20250902 01:59:16.638614 1520 log.cc:927] Time spent T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20250902 01:59:17.087100 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.090030 31067 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:17.096019 30924 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:17.086148 30926 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:16.769212 31722 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:17.105091 31039 leader_election.cc:336] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.997s (SENT)
I20250902 01:59:17.105718 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.086019 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.109292 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.625478 1366 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.624138 1405 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:17.172626 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.174190 31730 raft_consensus.cc:2408] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20250902 01:59:17.170019 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:17.169185 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 64129.5us
I20250902 01:59:17.169296 30923 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:17.129575 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.172524 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.175652 31075 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
W20250902 01:59:16.814610 1509 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.322s user 0.000s sys 0.001s
I20250902 01:59:17.175863 31725 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:16.639050 1475 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250902 01:59:16.934923 31721 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.021s sys 0.022s
I20250902 01:59:17.204099 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.220710 30919 raft_consensus.cc:2374] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:17.041566 1479 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:17.204754 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.179262 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.178918 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:16.639626 1386 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:17.042831 1438 consensus_meta.cc:287] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.595s user 0.006s sys 0.000s
I20250902 01:59:17.207314 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.100845 1451 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:17.213402 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.221657 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.179563 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:16.999395 1436 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.223748 31726 raft_consensus.cc:1273] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250902 01:59:16.718712 1465 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:16.680815 1498 raft_consensus.cc:2802] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:17.225106 31729 raft_consensus.cc:2391] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:17.173131 31727 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.006s sys 0.018s
I20250902 01:59:17.227407 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:17.131059 1468 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.087087 1292 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:16.819568 1448 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.228135 30920 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:17.176833 1464 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.015s sys 0.023s
I20250902 01:59:17.228860 31730 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:17.230029 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.212529 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.232815 31725 raft_consensus.cc:2408] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:17.232733 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:17.100687 1472 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:16.663755 1469 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.250780 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:17.248669 1464 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.019s sys 0.023s
I20250902 01:59:17.252696 1469 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:16.777907 1503 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:17.128790 31724 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
W20250902 01:59:17.224699 31721 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s user 0.026s sys 0.034s
I20250902 01:59:17.255183 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:17.256036 30924 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:17.258670 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.238286 1386 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:59:17.238211 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.012s sys 0.016s
W20250902 01:59:17.130653 1336 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.006s sys 0.043s
W20250902 01:59:17.228479 31723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.013s sys 0.023s
W20250902 01:59:17.239146 30996 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.009s sys 0.019s
W20250902 01:59:17.238801 31727 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.794s user 0.006s sys 0.018s
I20250902 01:59:17.213013 1482 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:16.663959 1430 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.221284 1402 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:17.244107 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.221118 1411 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:17.128991 1425 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
W20250902 01:59:17.270881 1526 log.cc:927] Time spent T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
W20250902 01:59:17.275050 1518 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.968s user 0.002s sys 0.003s
W20250902 01:59:17.275919 1524 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.461s user 0.000s sys 0.001s
I20250902 01:59:17.276229 1536 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:17.235738 1438 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.239285 1466 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.279688 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 465) took 1510 ms. Trace:
I20250902 01:59:17.281425 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:17.283905 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:17.305124 30917 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:17.294950 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:17.295676 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.856s user 0.012s sys 0.016s
I20250902 01:59:17.288574 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.298142 30919 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
W20250902 01:59:17.298782 30996 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s user 0.009s sys 0.020s
I20250902 01:59:17.238041 1465 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:17.246327 1292 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:17.265992 1448 consensus_queue.cc:1046] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20250902 01:59:17.304706 31000 rpcz_store.cc:276] 0902 01:59:15.769305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.769651 (+ 346us) service_pool.cc:225] Handling call
0902 01:59:17.279480 (+1509829us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001aefd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:17.279625 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267679616}
W20250902 01:59:17.296968 1336 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.994s user 0.009s sys 0.049s
I20250902 01:59:17.328094 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.237465 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:17.297210 31723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s user 0.013s sys 0.024s
W20250902 01:59:17.332942 1506 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.735s user 0.010s sys 0.000s
I20250902 01:59:17.333451 1506 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531) with higher term. Message: Invalid argument: T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 8. Current term is 9.
W20250902 01:59:17.334535 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:17.335887 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.336488 31000 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:17.337395 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.337476 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.264436 1498 raft_consensus.cc:695] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:17.311450 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.346665 31695 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:17.369238 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.370193 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:17.371448 30926 raft_consensus.cc:2374] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:17.371728 31071 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250902 01:59:17.373129 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.375560 31695 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:17.330938 31693 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:17.238338 1451 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.370463 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:17.343658 1272 consensus_meta.cc:287] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.833s user 0.010s sys 0.000s
I20250902 01:59:17.355541 31038 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:17.382272 30889 connection.cc:232] RPC call timeout handler was delayed by -1.9171743330000481s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20250902 01:59:17.383872 30889 connection.cc:232] RPC call timeout handler was delayed by -1.6750743329999978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19690000000000002s for a response
W20250902 01:59:17.384155 30889 connection.cc:232] RPC call timeout handler was delayed by -1.5715743330000578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
W20250902 01:59:17.384413 30889 connection.cc:232] RPC call timeout handler was delayed by -1.0243743329999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26920000000000005s for a response
I20250902 01:59:17.384572 31695 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:17.389019 30992 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 9.
I20250902 01:59:17.389928 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.391245 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.392724 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:17.393769 31072 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.003s sys 0.019s
W20250902 01:59:17.394172 31072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s user 0.003s sys 0.019s
I20250902 01:59:17.397305 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.398041 30923 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:17.402266 30889 leader_election.cc:400] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6.
I20250902 01:59:17.403246 30889 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:17.409955 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.410734 31069 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:17.412395 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.295642 1472 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:17.381064 1412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.002s sys 0.024s
I20250902 01:59:17.369383 1465 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:17.367626 1469 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:17.380235 1479 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:17.416325 31695 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:17.382828 1468 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:17.367940 1459 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:17.411752 1524 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20250902 01:59:17.295235 1405 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.305671 1411 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.422152 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.417145 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.437121 30917 raft_consensus.cc:2408] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250902 01:59:17.422701 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:17.423655 1471 consensus_meta.cc:287] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.053s user 0.010s sys 0.000s
I20250902 01:59:17.423877 30923 raft_consensus.cc:1273] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:17.438736 30924 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:17.410003 1527 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:17.419302 1426 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:17.413214 1484 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:17.425144 30994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.005s sys 0.017s
W20250902 01:59:17.415086 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 84249.6us
W20250902 01:59:17.441289 30994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s user 0.005s sys 0.017s
I20250902 01:59:17.441577 31693 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:17.349236 1536 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.373849 1436 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20250902 01:59:17.379518 1475 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:17.413434 1024 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:17.464715 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.004s sys 0.045s
W20250902 01:59:17.466707 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.004s sys 0.045s
I20250902 01:59:17.424041 1321 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:17.462297 1527 consensus_queue.cc:1035] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20250902 01:59:17.418140 1328 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:17.446652 1466 raft_consensus.cc:2802] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:17.433614 867 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.004s sys 0.020s
I20250902 01:59:17.425684 1468 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:17.411938 1272 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.414947 1398 consensus_queue.cc:1035] T 151f832cfd4e42d3a547a7cef6bb995f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20250902 01:59:17.436491 1538 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:17.412405 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:17.425786 1469 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:17.419864 1412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s user 0.002s sys 0.024s
W20250902 01:59:17.426326 30998 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.801s user 0.006s sys 0.016s
W20250902 01:59:17.540745 30998 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.916s user 0.006s sys 0.017s
W20250902 01:59:17.464165 1513 consensus_meta.cc:287] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.903s user 0.006s sys 0.003s
W20250902 01:59:17.479317 31001 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1915) took 1766 ms (client timeout 1986 ms). Trace:
W20250902 01:59:17.541671 31001 rpcz_store.cc:269] 0902 01:59:15.713114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.715356 (+ 2242us) service_pool.cc:225] Handling call
0902 01:59:17.479140 (+1763784us) spinlock_profiling.cc:243] Waited 91.1 ms on lock 0x7b5400146ed4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:17.479269 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209482496}
I20250902 01:59:17.435778 1459 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:17.439409 1424 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.016s sys 0.012s
I20250902 01:59:17.469695 1425 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:17.492481 1468 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
W20250902 01:59:17.519050 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:17.485415 867 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.004s sys 0.020s
I20250902 01:59:17.526690 30926 raft_consensus.cc:2391] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 4: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:17.476329 1328 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:17.578953 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.544180 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1857) took 1616 ms. Trace:
I20250902 01:59:17.427146 1531 consensus_queue.cc:1046] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20250902 01:59:17.484494 1386 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:17.447185 1471 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:17.551869 30920 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:17.468850 1321 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:17.578879 1400 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.067s user 0.022s sys 0.070s
I20250902 01:59:17.466792 1024 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.626756 1024 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:17.613466 1400 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s user 0.022s sys 0.071s
I20250902 01:59:17.577919 1292 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.493252 1466 raft_consensus.cc:695] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:17.579180 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 487091us
I20250902 01:59:17.492468 1537 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:17.580459 31066 rpcz_store.cc:276] 0902 01:59:15.928295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.928443 (+ 148us) service_pool.cc:225] Handling call
0902 01:59:16.124048 (+195605us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:16.124276 (+ 228us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:17.102547 (+978271us) log.cc:844] Serialized 14 byte log entry
0902 01:59:17.543723 (+441176us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:17.543795 (+ 72us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:17.543851 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.543917 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:17.544119 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":369290,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":1382,"thread_start_us":1418809,"threads_started":2}
W20250902 01:59:17.549994 1424 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.016s sys 0.013s
I20250902 01:59:17.610926 31074 raft_consensus.cc:604] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Received request to transfer leadership to TS 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:17.550201 1252 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:17.578915 522 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:17.515846 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.004s sys 0.025s
I20250902 01:59:17.628165 31728 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
W20250902 01:59:17.743026 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.026s user 0.004s sys 0.025s
I20250902 01:59:17.743976 522 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:17.744546 522 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:17.746737 522 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:17.750525 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 395076us
I20250902 01:59:17.755431 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:17.756124 31066 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:17.586138 1531 consensus_queue.cc:1230] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer 01f6731ce5044a6a948a5ae04da80e6f log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250902 01:59:17.637667 30918 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:17.684453 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:17.462769 1498 consensus_queue.cc:237] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.550522 1513 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:17.728943 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:17.583539 1328 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:17.585638 1430 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.614485 1451 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:17.805619 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 69808.3us
W20250902 01:59:17.805689 30922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.016s sys 0.020s
I20250902 01:59:17.544497 30924 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
W20250902 01:59:17.812369 30892 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.000s sys 0.013s
W20250902 01:59:17.817207 30922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.016s sys 0.020s
I20250902 01:59:17.817631 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1744) took 1286 ms. Trace:
W20250902 01:59:17.817857 30892 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.001s sys 0.013s
I20250902 01:59:17.818028 30924 rpcz_store.cc:276] 0902 01:59:16.530906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.393652 (+862746us) service_pool.cc:225] Handling call
0902 01:59:17.817554 (+423902us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:17.814405 31692 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.722s user 0.026s sys 0.013s
W20250902 01:59:17.818773 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 724849us
W20250902 01:59:17.818958 31692 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.726s user 0.026s sys 0.013s
W20250902 01:59:17.819330 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 726953us
I20250902 01:59:17.824914 31066 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:17.635802 1412 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:17.757123 1438 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:17.856866 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 565) took 1615 ms. Trace:
I20250902 01:59:17.863063 30922 rpcz_store.cc:276] 0902 01:59:16.241533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.241695 (+ 162us) service_pool.cc:225] Handling call
0902 01:59:16.273931 (+ 32236us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:16.274202 (+ 271us) raft_consensus.cc:1532] Early marking committed up to index 9
0902 01:59:17.100993 (+826791us) log.cc:844] Serialized 14 byte log entry
0902 01:59:17.856532 (+755539us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:17.856586 (+ 54us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:17.856633 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.856702 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:17.856813 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1054793,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":2652,"raft.queue_time_us":41075,"thread_start_us":1488126,"threads_started":2}
I20250902 01:59:17.857381 30892 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:17.865867 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 47426.4us
I20250902 01:59:17.638020 1527 consensus_queue.cc:1035] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.228s
I20250902 01:59:17.867630 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:17.579797 1425 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:17.878458 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 126437us
W20250902 01:59:17.878772 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:17.607664 1386 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:17.860514 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 417333us
I20250902 01:59:17.861214 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.891434 31001 raft_consensus.cc:2408] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250902 01:59:17.862586 31070 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:17.800045 1483 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:17.867228 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 432580us
I20250902 01:59:17.876305 31730 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
W20250902 01:59:17.859776 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 421694us
I20250902 01:59:17.893512 31693 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:17.894104 31695 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:17.888098 31692 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:17.856700 1412 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:17.856906 1537 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:17.901588 30889 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:17.903699 30889 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:17.908974 31072 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.009s sys 0.002s
W20250902 01:59:17.909632 31072 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.010s sys 0.003s
I20250902 01:59:17.864953 1538 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:17.867225 1531 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:17.877365 1472 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:17.872121 1411 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:17.908064 31695 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
W20250902 01:59:17.918109 31000 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.006s sys 0.006s
W20250902 01:59:17.926859 31000 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.527s user 0.007s sys 0.006s
I20250902 01:59:17.896117 867 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:17.934382 31067 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.007s sys 0.051s
I20250902 01:59:17.937381 30967 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:17.935779 1401 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.835s user 0.007s sys 0.023s
I20250902 01:59:17.945047 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:17.979267 31067 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.007s sys 0.051s
W20250902 01:59:17.963956 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 60323.3us
I20250902 01:59:17.965287 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:17.898509 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:17.981129 31071 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:17.993963 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.994892 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.995348 31071 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:17.996073 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.900493 1412 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:17.878135 1024 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20250902 01:59:17.978011 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.998644 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:18.000742 31001 raft_consensus.cc:2391] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:18.003546 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 66280.6us
W20250902 01:59:18.003917 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 110608us
W20250902 01:59:18.008893 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 46678.2us
W20250902 01:59:18.009758 31039 leader_election.cc:336] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.489s (SENT)
I20250902 01:59:17.895752 1506 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
W20250902 01:59:18.010763 31039 connection.cc:232] RPC call timeout handler was delayed by -1.0239014200000156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2806s for a response
W20250902 01:59:18.011878 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 103903us
I20250902 01:59:18.022260 31695 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
W20250902 01:59:18.012260 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:17.993224 1401 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.906s user 0.007s sys 0.023s
I20250902 01:59:18.031725 30892 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:18.028805 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:17.929054 1398 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:18.034847 30999 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 467) took 2259 ms (client timeout 1715 ms). Trace:
W20250902 01:59:18.049338 30999 rpcz_store.cc:269] 0902 01:59:15.775113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.775706 (+ 593us) service_pool.cc:225] Handling call
0902 01:59:18.034651 (+2258945us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000b5c54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:18.034752 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241366784}
I20250902 01:59:18.050500 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:18.034368 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 517) took 1520 ms. Trace:
I20250902 01:59:18.051748 31729 rpcz_store.cc:276] 0902 01:59:16.513547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.515646 (+ 2099us) service_pool.cc:225] Handling call
0902 01:59:18.034153 (+1518507us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54006dddd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:18.034305 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300640640}
I20250902 01:59:18.053011 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:18.053612 31729 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:17.982259 1472 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:18.047155 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.979655 1548 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:17.979245 1272 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.067653 30999 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:18.016085 31724 raft_consensus.cc:3058] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:18.037070 1501 log.cc:927] Time spent T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.796s user 0.002s sys 0.008s
I20250902 01:59:18.006669 1536 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.029240 1430 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:17.927826 1538 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.033707 1515 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:18.028990 1436 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:18.082033 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:17.995409 867 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.032291 1531 raft_consensus.cc:2747] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:18.040354 1469 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:18.066347 1543 log.cc:927] Time spent T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250902 01:59:18.048549 1459 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:18.070242 1535 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.526s user 0.000s sys 0.001s
I20250902 01:59:17.924203 1466 consensus_queue.cc:237] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.100760 1538 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.101238 1381 raft_consensus.cc:3053] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:18.101639 1381 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:18.102319 1381 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.103221 1381 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:18.079201 1541 log.cc:927] Time spent T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.497s user 0.000s sys 0.001s
I20250902 01:59:18.103963 1024 consensus_queue.cc:1035] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250902 01:59:18.104521 1544 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.238s user 0.000s sys 0.001s
I20250902 01:59:18.100756 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 472) took 2284 ms. Trace:
I20250902 01:59:18.105300 1024 consensus_queue.cc:1046] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:18.101284 1538 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250902 01:59:18.106483 1549 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.211s user 0.000s sys 0.000s
I20250902 01:59:18.106848 1538 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:18.077556 1321 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:18.107324 1515 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:18.094069 1436 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:18.107739 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 474) took 2215 ms. Trace:
I20250902 01:59:18.108026 1436 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:18.082296 1547 raft_consensus.cc:2802] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:18.108095 30994 rpcz_store.cc:276] 0902 01:59:15.892009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.513038 (+621029us) service_pool.cc:225] Handling call
0902 01:59:16.513298 (+ 260us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:16.523945 (+ 10647us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:16.523984 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:16.524502 (+ 518us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.447206 (+922704us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:17.447251 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.447317 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.107548 (+660231us) raft_consensus.cc:1683] finished
0902 01:59:18.107584 (+ 36us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.107691 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25067,"raft.queue_time_us":19130,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":9641,"thread_start_us":876615,"threads_started":1,"wal-append.queue_time_us":1085310}
I20250902 01:59:18.032163 1553 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:18.122660 1553 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.124610 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 966) took 3429 ms. Trace:
I20250902 01:59:18.105314 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:38794 (request call id 1130) took 2361 ms. Trace:
I20250902 01:59:18.083773 1272 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:18.094455 31001 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:18.107189 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 614) took 1575 ms. Trace:
I20250902 01:59:18.105172 30996 rpcz_store.cc:276] 0902 01:59:15.816037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.349019 (+532982us) service_pool.cc:225] Handling call
0902 01:59:16.349262 (+ 243us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:16.358184 (+ 8922us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:16.358225 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:16.358625 (+ 400us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.365613 (+1006988us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:17.365663 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.373016 (+ 7353us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.100558 (+727542us) raft_consensus.cc:1683] finished
0902 01:59:18.100578 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.100688 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8616,"raft.queue_time_us":9569,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":1003178,"threads_started":1,"wal-append.queue_time_us":1103942}
I20250902 01:59:18.133773 30925 rpcz_store.cc:276] 0902 01:59:15.744198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.744933 (+ 735us) service_pool.cc:225] Handling call
0902 01:59:18.104811 (+2359878us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b5400337a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b4cd0 00007fc602138eea 00007fc60709160e 00007fc60072ecc7 00007fc60072ec52 00007fc60072eb81 00007fc60072eafc 00007fc60072ddb2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:18.105226 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4307796736}
I20250902 01:59:18.134850 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.087350 1471 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.133738 31068 rpcz_store.cc:276] 0902 01:59:16.531329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.718485 (+187156us) service_pool.cc:225] Handling call
0902 01:59:16.718888 (+ 403us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:16.719266 (+ 378us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:16.719301 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.129801 (+410500us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.566173 (+436372us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:17.566222 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.566329 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.107002 (+540673us) raft_consensus.cc:1683] finished
0902 01:59:18.107020 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.107134 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":44727,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":1616,"thread_start_us":335005,"threads_started":1,"wal-append.queue_time_us":820183}
I20250902 01:59:18.093014 1536 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:18.107249 1538 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:18.105777 1552 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20250902 01:59:18.083500 1430 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:18.099339 1540 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
I20250902 01:59:18.124756 1553 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:18.131876 1542 log.cc:927] Time spent T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20250902 01:59:18.122969 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.135666 30996 raft_consensus.cc:2366] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:18.139571 31068 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:18.151387 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.166172 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.167493 30996 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:18.069659 1548 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.146468 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 461) took 4222 ms. Trace:
I20250902 01:59:18.168399 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.149120 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.092895 1292 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.166891 30994 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:18.126072 1547 raft_consensus.cc:695] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:18.135577 31723 rpcz_store.cc:276] 0902 01:59:14.695710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.401511 (+705801us) service_pool.cc:225] Handling call
0902 01:59:15.401890 (+ 379us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:16.445257 (+1043367us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:16.452341 (+ 7084us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:16.477182 (+ 24841us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.376394 (+899212us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:17.379721 (+ 3327us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.124142 (+744421us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b540062e154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:18.124196 (+ 54us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.124253 (+ 57us) raft_consensus.cc:1683] finished
0902 01:59:18.124265 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.124551 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":512904,"raft.queue_time_us":555267,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3864,"spinlock_wait_cycles":927114624,"thread_start_us":826921,"threads_started":1,"wal-append.queue_time_us":987712}
I20250902 01:59:18.168251 30993 rpcz_store.cc:276] 0902 01:59:13.918813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:13.919160 (+ 347us) service_pool.cc:225] Handling call
0902 01:59:13.919501 (+ 341us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:13.980822 (+ 61321us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:13.980863 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:13.996539 (+ 15676us) log.cc:844] Serialized 26 byte log entry
0902 01:59:15.723056 (+1726517us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:15.723092 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:15.723189 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.036424 (+2313235us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:18.132880 (+ 96456us) raft_consensus.cc:1683] finished
0902 01:59:18.132904 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.141508 (+ 8604us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8649,"raft.queue_time_us":8769,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":60190,"spinlock_wait_cycles":9976576,"thread_start_us":1630850,"threads_started":1,"wal-append.queue_time_us":1864268}
I20250902 01:59:18.107916 1515 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:18.175206 1570 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250902 01:59:18.150672 1430 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:18.069579 1472 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.141279 1513 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.188889 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1925) took 1545 ms. Trace:
I20250902 01:59:18.225461 30994 rpcz_store.cc:276] 0902 01:59:16.643857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.187987 (+1544130us) service_pool.cc:225] Handling call
0902 01:59:18.188326 (+ 339us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.188509 (+ 183us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.188541 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:18.188594 (+ 53us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:18.188634 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.188698 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.188826 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:18.145946 1538 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:18.147617 1531 raft_consensus.cc:3053] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:18.066145 1398 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.216511 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.250707 31001 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:18.131361 1428 consensus_queue.cc:1035] T d0b2dd8f5f814a7d8b48a74020055750 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20250902 01:59:18.164979 1506 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:18.219673 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 469) took 2439 ms. Trace:
W20250902 01:59:18.151103 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 204490us
W20250902 01:59:18.238402 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 218867us
I20250902 01:59:18.142321 1272 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.255944 30995 rpcz_store.cc:276] 0902 01:59:15.780112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.780232 (+ 120us) service_pool.cc:225] Handling call
0902 01:59:15.780503 (+ 271us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:17.103775 (+1323272us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.103814 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.104255 (+ 441us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.544730 (+440475us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:17.544776 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.039443 (+494667us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54000d66d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:18.039500 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.219351 (+179851us) raft_consensus.cc:1683] finished
0902 01:59:18.219371 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.219603 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32116,"raft.queue_time_us":14241,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":16466,"spinlock_wait_cycles":1171153792,"thread_start_us":1239768,"threads_started":2,"wal-append.queue_time_us":810857}
I20250902 01:59:18.217265 1515 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:18.261158 30967 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:18.262842 1292 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:18.263204 1292 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:18.263581 1292 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:18.264084 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:18.217180 1568 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:18.098344 1537 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.280040 1580 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:18.296831 1580 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.299358 1580 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.273018 1568 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:18.286532 30996 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:18.300388 1568 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:18.304823 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:18.304888 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.288789 1472 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.292735 1398 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.293483 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 975) took 2063 ms. Trace:
I20250902 01:59:18.250202 1531 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:18.268468 31729 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 9.
I20250902 01:59:18.322680 31727 rpcz_store.cc:276] 0902 01:59:16.229650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.229858 (+ 208us) service_pool.cc:225] Handling call
0902 01:59:16.260041 (+ 30183us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:17.297438 (+1037397us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.297472 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.303234 (+ 5762us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.758603 (+455369us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:17.758639 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.758708 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.293251 (+534543us) raft_consensus.cc:1683] finished
0902 01:59:18.293270 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.293420 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":184323,"raft.queue_time_us":219515,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":6470,"spinlock_wait_cycles":12160,"thread_start_us":1231710,"threads_started":2,"wal-append.queue_time_us":849224}
W20250902 01:59:18.296205 1413 consensus_meta.cc:287] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.121s user 0.008s sys 0.001s
W20250902 01:59:18.324920 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 293286us
I20250902 01:59:18.327469 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:18.349454 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:18.328676 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 306487us
I20250902 01:59:18.274349 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:18.273294 1425 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.354281 30892 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:18.325518 1570 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20250902 01:59:18.358215 30965 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.577s (SENT)
W20250902 01:59:18.366780 30965 leader_election.cc:336] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.594s (SENT)
I20250902 01:59:18.368139 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:18.391894 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1941) took 1874 ms. Trace:
I20250902 01:59:18.392145 30926 raft_consensus.cc:1273] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20250902 01:59:18.368791 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.374249 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:18.373585 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 485) took 1819 ms. Trace:
W20250902 01:59:18.376130 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 410911us
I20250902 01:59:18.379861 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:18.372287 1461 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.080s user 0.028s sys 0.013s
I20250902 01:59:18.266275 1506 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.386910 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1931) took 1006 ms. Trace:
I20250902 01:59:18.391939 30965 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:18.367741 1398 raft_consensus.cc:2747] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:18.434969 31000 rpcz_store.cc:276] 0902 01:59:16.554618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.372069 (+817451us) service_pool.cc:225] Handling call
0902 01:59:17.372329 (+ 260us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.372512 (+ 183us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:17.927959 (+555447us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.373150 (+445191us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:18.373214 (+ 64us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:18.373275 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.373358 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.373491 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":526075,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":109,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":14526,"thread_start_us":972549,"threads_started":2}
I20250902 01:59:18.494572 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:18.495383 31000 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:18.449484 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:18.498075 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:18.498853 31068 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:18.427956 31723 raft_consensus.cc:2391] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:18.501484 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1927) took 1857 ms. Trace:
I20250902 01:59:18.492343 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:18.504143 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 150189us
W20250902 01:59:18.502043 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 135309us
I20250902 01:59:18.503945 30994 rpcz_store.cc:276] 0902 01:59:16.644374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.237918 (+1593544us) service_pool.cc:225] Handling call
0902 01:59:18.238238 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.238444 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:18.257045 (+ 18601us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.501195 (+244150us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:18.501242 (+ 47us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:18.501286 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.501342 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.501440 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7849,"prepare.queue_time_us":26455,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":31,"raft.queue_time_us":69515,"thread_start_us":174296,"threads_started":1}
I20250902 01:59:18.499930 30964 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:18.505926 30965 leader_election.cc:336] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.650s (SENT)
I20250902 01:59:18.506347 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:18.502084 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:18.507302 30965 leader_election.cc:336] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.658s (SENT)
I20250902 01:59:18.506906 30964 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:18.508286 1569 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:18.508805 31724 raft_consensus.cc:2466] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:18.509547 30964 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:18.508829 31727 raft_consensus.cc:2391] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:18.511786 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:18.512028 30965 leader_election.cc:336] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.036s (SENT)
W20250902 01:59:18.513243 30965 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.069s (SENT) [suppressed 26 similar messages]
I20250902 01:59:18.512454 30994 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:18.398181 1578 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:18.492774 1461 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.201s user 0.028s sys 0.023s
I20250902 01:59:18.516070 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.525511 30889 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:18.518601 30965 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:18.353163 1413 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.375579 1472 consensus_queue.cc:1046] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.127s
I20250902 01:59:18.526611 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:18.538456 30872 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 92f16b8a15144fcf8807608bdee71123
W20250902 01:59:18.540575 30872 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 92f16b8a15144fcf8807608bdee71123
I20250902 01:59:18.541471 30965 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:18.542757 30965 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:18.547111 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:18.580333 31000 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:18.511626 1536 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:18.346088 1547 consensus_queue.cc:237] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.392308 30917 rpcz_store.cc:276] 0902 01:59:16.516987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.444090 (+927103us) service_pool.cc:225] Handling call
0902 01:59:17.462796 (+ 18706us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:17.463074 (+ 278us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.463103 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.463517 (+ 414us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.917732 (+454215us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:17.963379 (+ 45647us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.325493 (+362114us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.325584 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.391699 (+ 66115us) raft_consensus.cc:1683] finished
0902 01:59:18.391724 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.391844 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43866,"prepare.queue_time_us":374002,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":304415,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":158,"thread_start_us":777022,"threads_started":3,"wal-append.queue_time_us":562998}
I20250902 01:59:18.667304 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:18.502562 1381 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:18.652081 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 142594us
W20250902 01:59:18.606683 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.668694 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:18.673065 30964 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:18.673373 31000 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 8: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:18.563220 30892 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:18.619961 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:18.675046 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 148505us
I20250902 01:59:18.675840 30889 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:18.677511 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.527833 1527 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:18.538101 31724 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:18.685181 1527 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.543617 1471 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:18.689302 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1930) took 1311 ms. Trace:
W20250902 01:59:18.670835 1555 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.416s user 0.000s sys 0.001s
W20250902 01:59:18.652397 1465 consensus_meta.cc:287] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.229s user 0.005s sys 0.010s
I20250902 01:59:18.544289 1580 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:18.675590 1272 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.529927 1566 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:18.545122 1584 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
I20250902 01:59:18.508850 1569 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.773303 1612 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:18.773680 1583 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:18.585224 1436 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:18.747684 1562 log.cc:927] Time spent T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.471s user 0.000s sys 0.001s
I20250902 01:59:18.708081 1381 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:18.694701 1553 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:18.547422 867 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.581593 1548 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:18.529798 1590 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:18.513253 1531 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:18.686807 1359 consensus_meta.cc:287] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.558s user 0.008s sys 0.003s
I20250902 01:59:18.690021 1536 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:18.828644 1536 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:18.828603 1359 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.829119 1536 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:18.793802 1580 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:18.800010 1566 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.832440 1580 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:18.832803 1580 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:18.584889 1430 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:18.815847 1602 log.cc:927] Time spent T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.311s user 0.000s sys 0.001s
W20250902 01:59:18.831713 1603 log.cc:927] Time spent T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20250902 01:59:18.831902 1359 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.529337 1578 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.836503 1566 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:18.842355 1468 consensus_meta.cc:287] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 1.255s user 0.009s sys 0.005s
I20250902 01:59:18.842820 1468 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.845245 1468 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:18.846487 1412 consensus_meta.cc:287] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.816s user 0.007s sys 0.009s
W20250902 01:59:18.687294 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:18.768918 31001 rpcz_store.cc:276] 0902 01:59:17.377806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.255301 (+877495us) service_pool.cc:225] Handling call
0902 01:59:18.304195 (+ 48894us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.304437 (+ 242us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:18.346985 (+ 42548us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.688776 (+341791us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:18.688839 (+ 63us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:18.688931 (+ 92us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.689073 (+ 142us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.689218 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16053,"prepare.queue_time_us":42776,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"thread_start_us":325231,"threads_started":2}
W20250902 01:59:18.794060 31730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.012s sys 0.025s
I20250902 01:59:18.790474 1465 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.837539 1527 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:18.805191 1569 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:18.847023 1608 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20250902 01:59:19.201064 1465 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:19.093950 1561 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20250902 01:59:19.058429 1566 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:19.073523 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:19.080410 1364 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.086028 1386 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:18.993010 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.206398 31725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 532) took 1340 ms. Trace:
I20250902 01:59:19.206928 31725 rpcz_store.cc:276] 0902 01:59:17.866156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.866283 (+ 127us) service_pool.cc:225] Handling call
0902 01:59:19.072974 (+1206691us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b58001e3998. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:19.206311 (+133337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8359168}
W20250902 01:59:19.208257 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:18.991935 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 428557us
I20250902 01:59:19.210160 31725 raft_consensus.cc:1238] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250902 01:59:19.074342 1424 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:19.001907 31073 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:19.072203 522 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:19.198907 31071 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:19.196424 31730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.048s user 0.018s sys 0.029s
I20250902 01:59:19.049410 1413 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:19.008402 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 332484us
I20250902 01:59:19.047230 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1746) took 2515 ms. Trace:
I20250902 01:59:19.059156 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 484) took 2539 ms. Trace:
I20250902 01:59:19.196787 1412 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:18.993003 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1946) took 2474 ms. Trace:
I20250902 01:59:19.214238 31072 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1848) took 3450 ms. Trace:
I20250902 01:59:19.070513 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 523) took 2555 ms. Trace:
W20250902 01:59:19.219777 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:18.895617 1604 log.cc:927] Time spent T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.119s user 0.002s sys 0.000s
I20250902 01:59:19.221295 1376 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.222175 1424 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:18.940519 1590 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:19.088855 1564 consensus_queue.cc:1035] T 4120ecbc46e34f93bb86004798318c08 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.139s
I20250902 01:59:18.918402 1580 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.057617 1553 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:19.042958 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1925) took 2536 ms. Trace:
W20250902 01:59:19.029575 1405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.904s user 0.000s sys 0.030s
I20250902 01:59:19.069422 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 625) took 1715 ms. Trace:
W20250902 01:59:19.229391 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.061177 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 613) took 2529 ms. Trace:
I20250902 01:59:19.068403 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1726) took 1126 ms. Trace:
W20250902 01:59:18.926277 1568 consensus_meta.cc:287] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.625s user 0.008s sys 0.000s
I20250902 01:59:18.992136 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:19.257552 30918 rpcz_store.cc:276] 0902 01:59:16.506243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.549972 (+ 43729us) service_pool.cc:225] Handling call
0902 01:59:16.550209 (+ 237us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.798449 (+1248240us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.042281 (+1243832us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400337a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.042728 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2500656384}
W20250902 01:59:18.897815 1515 consensus_meta.cc:287] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.621s user 0.008s sys 0.000s
W20250902 01:59:19.065380 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 801243us
I20250902 01:59:19.000715 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:18.913486 1538 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.686s user 0.004s sys 0.004s
I20250902 01:59:19.230309 31722 rpcz_store.cc:276] 0902 01:59:16.515005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.595693 (+ 80688us) service_pool.cc:225] Handling call
0902 01:59:16.595952 (+ 259us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.173341 (+577389us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.069963 (+1896622us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54006a9fd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.070293 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":393901952}
I20250902 01:59:19.065738 1511 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:19.050984 30992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 1924) took 2407 ms (client timeout 1588 ms). Trace:
I20250902 01:59:19.073758 1336 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:19.074731 1411 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:19.030730 1430 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:19.264338 1405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.169s user 0.004s sys 0.038s
W20250902 01:59:19.283901 30992 rpcz_store.cc:269] 0902 01:59:16.643537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.281133 (+637596us) service_pool.cc:225] Handling call
0902 01:59:19.050505 (+1769372us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b54002bdc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.050728 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99322240}
I20250902 01:59:19.285312 1561 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:19.217449 30998 rpcz_store.cc:276] 0902 01:59:16.519205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.593351 (+ 74146us) service_pool.cc:225] Handling call
0902 01:59:16.593636 (+ 285us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:16.601975 (+ 8339us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:17.541094 (+939119us) log.cc:844] Serialized 14 byte log entry
0902 01:59:17.935169 (+394075us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:17.935844 (+ 675us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:17.936068 (+ 224us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.058607 (+1122539us) spinlock_profiling.cc:243] Waited 8.1 ms on lock 0x7b54000d6bd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.058680 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.058945 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1193233,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":12254,"raft.queue_time_us":105995,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":18669,"spinlock_wait_cycles":18639616,"thread_start_us":1309202,"threads_started":2,"wal-append.queue_time_us":583939,"wal-append.run_cpu_time_us":1230,"wal-append.run_wall_time_us":1066696}
I20250902 01:59:19.260624 31070 rpcz_store.cc:276] 0902 01:59:16.531114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.639937 (+108823us) service_pool.cc:225] Handling call
0902 01:59:16.661797 (+ 21860us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.896174 (+1234377us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.060571 (+1164397us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400200654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.060910 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2761371008}
I20250902 01:59:19.030088 1436 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.072837 1359 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:19.263487 1553 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:19.221293 522 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:19.286036 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1928) took 2777 ms. Trace:
W20250902 01:59:19.294951 1375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.880s user 0.031s sys 0.056s
I20250902 01:59:19.230787 30923 rpcz_store.cc:276] 0902 01:59:16.518077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.445434 (+927357us) service_pool.cc:225] Handling call
0902 01:59:17.465682 (+ 20248us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:17.935089 (+469407us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.935287 (+ 198us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.938858 (+ 3571us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.950905 (+ 12047us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:17.951005 (+ 100us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:17.951156 (+ 151us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:18.992363 (+1041207us) raft_consensus.cc:1683] finished
0902 01:59:18.992429 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.992792 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":162021,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":479257,"threads_started":2,"wal-append.queue_time_us":128881}
I20250902 01:59:19.304736 1506 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.306847 1578 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.306392 522 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.312320 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1923) took 2669 ms. Trace:
I20250902 01:59:19.313477 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 489) took 1349 ms. Trace:
I20250902 01:59:19.312703 30999 rpcz_store.cc:276] 0902 01:59:16.643249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.203568 (+1560319us) service_pool.cc:225] Handling call
0902 01:59:18.208566 (+ 4998us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:18.219949 (+ 11383us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.220009 (+ 60us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:18.220661 (+ 652us) log.cc:844] Serialized 26 byte log entry
0902 01:59:18.431237 (+210576us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:18.431302 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.431389 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.311858 (+880469us) raft_consensus.cc:1683] finished
0902 01:59:19.311896 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.312125 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10787,"raft.queue_time_us":10939,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":13498,"thread_start_us":158105,"threads_started":1,"wal-append.queue_time_us":521552}
I20250902 01:59:19.262835 31066 rpcz_store.cc:276] 0902 01:59:17.942583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.942949 (+ 366us) service_pool.cc:225] Handling call
0902 01:59:17.943270 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.943475 (+ 205us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:17.943813 (+ 338us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.067697 (+1123884us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.067798 (+ 101us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:19.067930 (+ 132us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.068028 (+ 98us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.068217 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1118222,"prepare.queue_time_us":36799,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":1126369,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":222,"spinlock_wait_cycles":27012352,"thread_start_us":5061,"threads_started":1,"wal-append.queue_time_us":337399}
I20250902 01:59:19.313850 30996 rpcz_store.cc:276] 0902 01:59:17.964098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.168196 (+204098us) service_pool.cc:225] Handling call
0902 01:59:19.313137 (+1144941us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002ad0d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.313303 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269967744}
I20250902 01:59:19.307780 1538 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.309198 30994 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:19.299499 30921 rpcz_store.cc:276] 0902 01:59:16.508544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.513792 (+ 5248us) service_pool.cc:225] Handling call
0902 01:59:16.563546 (+ 49754us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:16.630920 (+ 67374us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.285576 (+2654656us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b540034f8d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.285852 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40030080}
W20250902 01:59:19.309979 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 215974us
I20250902 01:59:19.310582 30917 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 3.
W20250902 01:59:19.301630 1375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.948s user 0.031s sys 0.056s
I20250902 01:59:19.298640 1553 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:19.311916 1411 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.309062 1336 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.304299 1359 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.315339 30999 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250902 01:59:19.319365 30996 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:19.322953 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.323143 1586 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:19.324635 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:19.325423 1336 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.325296 1561 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.326114 31069 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:19.326282 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:19.327175 31729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.000s sys 0.013s
I20250902 01:59:19.089345 1513 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.215s
W20250902 01:59:19.311602 1443 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.270s user 0.008s sys 0.004s
I20250902 01:59:19.259737 31067 rpcz_store.cc:276] 0902 01:59:17.354367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.354475 (+ 108us) service_pool.cc:225] Handling call
0902 01:59:17.368114 (+ 13639us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.368293 (+ 179us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:17.409901 (+ 41608us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.068846 (+1658945us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.068948 (+ 102us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:19.069022 (+ 74us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.069104 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.069278 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1113716,"prepare.queue_time_us":41822,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2232,"raft.queue_time_us":1092895,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":157,"thread_start_us":572838,"threads_started":1,"wal-append.queue_time_us":682533,"wal-append.run_cpu_time_us":1293,"wal-append.run_wall_time_us":1124825}
I20250902 01:59:19.306418 31038 leader_election.cc:400] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
W20250902 01:59:19.317955 1255 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.842s user 0.009s sys 0.016s
I20250902 01:59:19.332897 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1994) took 1841 ms. Trace:
I20250902 01:59:19.333282 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1995) took 1842 ms. Trace:
W20250902 01:59:19.342087 31729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s user 0.000s sys 0.014s
I20250902 01:59:19.217145 30924 rpcz_store.cc:276] 0902 01:59:16.531329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.864658 (+1333329us) service_pool.cc:225] Handling call
0902 01:59:17.887268 (+ 22610us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:18.005745 (+118477us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.005773 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:18.006370 (+ 597us) log.cc:844] Serialized 52 byte log entry
0902 01:59:18.374083 (+367713us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:18.374122 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.046377 (+672255us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b54001b2454. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.046487 (+ 110us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.046579 (+ 92us) raft_consensus.cc:1683] finished
0902 01:59:19.046606 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.047036 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79815,"raft.queue_time_us":132791,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":6020,"spinlock_wait_cycles":106645248,"thread_start_us":299470,"threads_started":1,"wal-append.queue_time_us":611375}
I20250902 01:59:19.230005 31072 rpcz_store.cc:276] 0902 01:59:15.735779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.735922 (+ 143us) service_pool.cc:225] Handling call
0902 01:59:15.740297 (+ 4375us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:15.740470 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:16.552226 (+811756us) log.cc:844] Serialized 14 byte log entry
0902 01:59:17.924908 (+1372682us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.064517 (+1139609us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b18004c4080. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff14665a 00007fc608617409 00007fc5ff007af0 00007fc5ff1cb894 00007fc5ff1cb805 00007fc602128020 00007fc60213a6c6 00007fc60214b237 00007fc60214b1b8
0902 01:59:19.064936 (+ 419us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.065027 (+ 91us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.065169 (+ 142us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.102798 (+ 37629us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.186552 (+ 83754us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7305,"prepare.queue_time_us":733623,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":1491962,"raft.run_cpu_time_us":430,"raft.run_wall_time_us":432,"spinlock_wait_cycles":99573120,"thread_start_us":2158996,"threads_started":3,"wal-append.queue_time_us":1032869,"wal-append.run_cpu_time_us":2691,"wal-append.run_wall_time_us":1613102}
I20250902 01:59:19.344691 30996 rpcz_store.cc:276] 0902 01:59:17.487917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.314842 (+1826925us) service_pool.cc:225] Handling call
0902 01:59:19.315341 (+ 499us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.328244 (+ 12903us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.329991 (+ 1747us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.343531 30999 rpcz_store.cc:276] 0902 01:59:17.487683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.314298 (+1826615us) service_pool.cc:225] Handling call
0902 01:59:19.314783 (+ 485us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.328115 (+ 13332us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.329534 (+ 1419us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:19.340097 1443 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.385s user 0.010s sys 0.004s
W20250902 01:59:19.345533 1255 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.991s user 0.009s sys 0.017s
I20250902 01:59:18.956885 1536 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.356698 31693 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:19.357482 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51407.3us
I20250902 01:59:19.358458 31038 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:19.358876 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2001) took 1863 ms. Trace:
I20250902 01:59:19.359795 1568 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.360730 1564 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:19.361187 1564 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:19.361567 1564 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:19.362473 1568 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.362599 30921 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:19.364634 1424 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.365213 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:19.366050 30923 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:19.366226 1538 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.367866 1412 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:19.369364 31000 raft_consensus.cc:1238] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:19.370023 31001 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
W20250902 01:59:19.371511 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.372669 1580 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.373265 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1866) took 1382 ms. Trace:
I20250902 01:59:19.371088 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:19.370074 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1996) took 1881 ms. Trace:
W20250902 01:59:19.366335 1573 log.cc:927] Time spent T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 1.086s user 0.000s sys 0.001s
I20250902 01:59:19.359349 30992 rpcz_store.cc:276] 0902 01:59:17.495317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.357665 (+1862348us) service_pool.cc:225] Handling call
0902 01:59:19.358062 (+ 397us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.358497 (+ 435us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.358522 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.358567 (+ 45us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:19.358594 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.358663 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.358795 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.369315 1536 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.374656 31000 rpcz_store.cc:276] 0902 01:59:17.488185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.352270 (+1864085us) service_pool.cc:225] Handling call
0902 01:59:19.368998 (+ 16728us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.369835 (+ 837us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.370012 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:19.373916 31073 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 608) took 2841 ms (client timeout 1581 ms). Trace:
I20250902 01:59:19.373626 31071 rpcz_store.cc:276] 0902 01:59:17.991054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.030856 (+ 39802us) service_pool.cc:225] Handling call
0902 01:59:18.031152 (+ 296us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.219005 (+1187853us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.372973 (+153968us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400020dd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.373191 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2684813056}
I20250902 01:59:19.354245 1537 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:19.388085 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:19.385032 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.020s sys 0.044s
W20250902 01:59:19.378870 31073 rpcz_store.cc:269] 0902 01:59:16.529738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.529861 (+ 123us) service_pool.cc:225] Handling call
0902 01:59:19.371287 (+2841426us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b5400020154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.371455 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5565866240}
W20250902 01:59:19.384435 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.009s sys 0.020s
W20250902 01:59:19.391731 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.020s sys 0.045s
I20250902 01:59:19.392684 31067 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:19.394256 1514 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:19.394711 1448 raft_consensus.cc:3053] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:19.394869 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.395251 1448 raft_consensus.cc:738] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:19.395650 31067 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:59:19.393071 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.009s sys 0.020s
I20250902 01:59:19.376238 1412 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:19.399497 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:19.405155 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 50924.3us
I20250902 01:59:19.395033 1514 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.407743 31693 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:19.408327 1328 consensus_meta.cc:287] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 1.520s user 0.006s sys 0.025s
I20250902 01:59:19.408694 1328 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.409529 1448 consensus_queue.cc:260] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.410558 1328 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.411383 31726 raft_consensus.cc:1238] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:19.395792 1586 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.411520 31072 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:19.413097 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 534) took 1537 ms. Trace:
I20250902 01:59:19.412470 1359 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:19.413475 31726 rpcz_store.cc:276] 0902 01:59:17.875475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.877511 (+ 2036us) service_pool.cc:225] Handling call
0902 01:59:17.877836 (+ 325us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.411851 (+1534015us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.412851 (+ 1000us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54005eef54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.413027 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3526208768}
I20250902 01:59:19.410629 1448 raft_consensus.cc:3058] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.419239 1479 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:19.419832 1527 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:19.419865 1479 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:19.420342 1605 log.cc:927] Time spent T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.572s user 0.011s sys 0.004s
I20250902 01:59:19.427702 31693 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:19.420302 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:19.426290 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 883598us
I20250902 01:59:19.397321 1579 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:19.429759 31071 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:19.457764 1411 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:19.457862 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 784885us
I20250902 01:59:19.466401 30918 raft_consensus.cc:1238] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
W20250902 01:59:19.483459 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 55949.4us
I20250902 01:59:19.486555 1595 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:19.487810 1466 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250902 01:59:19.489204 30964 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:19.490553 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 167851us
I20250902 01:59:19.491494 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 967) took 4787 ms. Trace:
I20250902 01:59:19.491905 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.491971 31721 rpcz_store.cc:276] 0902 01:59:14.704181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.548895 (+844714us) service_pool.cc:225] Handling call
0902 01:59:15.549246 (+ 351us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:16.229756 (+680510us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:16.235780 (+ 6024us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:16.288789 (+ 53009us) log.cc:844] Serialized 52 byte log entry
0902 01:59:17.254291 (+965502us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:17.254640 (+ 349us) log.cc:844] Serialized 14 byte log entry
0902 01:59:17.276158 (+ 21518us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.122894 (+846736us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540010b654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:18.122944 (+ 50us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.490507 (+1367563us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b540010be20. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020bd54e 00007fc6020c8060 00007fc6020c8008 00007fc6020c7f88 00007fc6020c7f43 00007fc6020c71cc 00007fc607f02336 00007fc607efcb88 00007fc602103e5b 00007fc602106950
0902 01:59:19.490960 (+ 453us) raft_consensus.cc:1683] finished
0902 01:59:19.490983 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.491295 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11765,"prepare.queue_time_us":895334,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":446133,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":66266,"spinlock_wait_cycles":608388736,"thread_start_us":965098,"threads_started":1,"wal-append.queue_time_us":979694}
I20250902 01:59:19.494494 1579 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:19.495810 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 112217us
I20250902 01:59:19.495505 1579 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:19.498824 30892 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:19.500234 1566 consensus_queue.cc:1046] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:59:19.501334 1601 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:19.504464 1514 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.505527 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 663) took 2417 ms. Trace:
W20250902 01:59:19.505878 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.506791 1405 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:18.438930 30995 rpcz_store.cc:276] 0902 01:59:17.380228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.258075 (+877847us) service_pool.cc:225] Handling call
0902 01:59:18.258367 (+ 292us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.258565 (+ 198us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:18.280510 (+ 21945us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.386549 (+106039us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:18.386596 (+ 47us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:18.386632 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.386709 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:18.386842 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6100,"prepare.queue_time_us":20632,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1797,"thread_start_us":105704,"threads_started":1}
I20250902 01:59:19.507356 1405 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:19.469043 1412 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.508805 1579 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.509610 1412 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.509984 1561 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.444850 1527 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.510664 1359 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:19.510739 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 983) took 1418 ms. Trace:
I20250902 01:59:19.491400 1498 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:19.519296 31724 rpcz_store.cc:276] 0902 01:59:18.092584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.509535 (+416951us) service_pool.cc:225] Handling call
0902 01:59:18.515297 (+ 5762us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.684778 (+169481us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.510375 (+825597us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b54006ad450. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.510566 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3295616}
I20250902 01:59:19.520141 1498 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:19.520591 1498 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:19.468493 1515 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.508834 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1935) took 1610 ms. Trace:
I20250902 01:59:19.507974 1405 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:19.499140 31039 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.513814 31038 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:19.448101 1537 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.514020 1527 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.515111 1252 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:19.538887 1252 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:19.505883 31728 rpcz_store.cc:276] 0902 01:59:17.087896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.088023 (+ 127us) service_pool.cc:225] Handling call
0902 01:59:17.177377 (+ 89354us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.742730 (+565353us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.505100 (+1762370us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54004128d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.505365 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1036374400}
I20250902 01:59:19.540879 1479 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.544266 1475 consensus_queue.cc:1046] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:19.544741 1411 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:19.544977 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:19.545322 31038 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:19.544309 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 45325us
I20250902 01:59:19.546288 1412 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:19.546908 1412 raft_consensus.cc:695] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:19.547233 31039 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:19.548029 1412 consensus_queue.cc:237] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.548964 1515 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.550107 1336 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250902 01:59:19.518044 1585 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 1.236s user 0.001s sys 0.000s
I20250902 01:59:19.523837 30995 rpcz_store.cc:276] 0902 01:59:17.898720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.507798 (+1609078us) service_pool.cc:225] Handling call
0902 01:59:19.508169 (+ 371us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.508473 (+ 304us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.508495 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.508545 (+ 50us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:19.508570 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.508612 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.508693 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.511345 1359 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:19.537544 30923 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
W20250902 01:59:19.550904 1292 consensus_meta.cc:287] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.287s user 0.012s sys 0.008s
I20250902 01:59:19.556000 1411 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:19.545331 1475 raft_consensus.cc:3053] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Stepping down as leader of term 7
W20250902 01:59:19.557916 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 66264.5us
I20250902 01:59:19.558241 1475 raft_consensus.cc:738] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:19.558873 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.559382 1475 consensus_queue.cc:260] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.560038 1024 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:19.560362 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1949) took 3041 ms. Trace:
I20250902 01:59:19.560741 30926 rpcz_store.cc:276] 0902 01:59:16.518673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.581160 (+1062487us) service_pool.cc:225] Handling call
0902 01:59:18.391845 (+810685us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.392494 (+ 649us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.559974 (+1167480us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54002a0dd0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.560174 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1756457728}
W20250902 01:59:19.565474 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 76465.2us
I20250902 01:59:19.552623 1479 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.564672 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:19.587834 1500 raft_consensus.cc:2802] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:19.588532 30965 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:19.561103 1475 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:19.554445 1359 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.585479 1527 raft_consensus.cc:2747] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:19.563918 1537 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.606621 31071 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250902 01:59:19.593360 1500 raft_consensus.cc:695] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [term 8 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:19.608736 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 110037us
I20250902 01:59:19.610981 31693 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:19.598428 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.608498 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 574) took 1089 ms. Trace:
I20250902 01:59:19.618078 30924 rpcz_store.cc:276] 0902 01:59:18.515076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.392101 (+877025us) service_pool.cc:225] Handling call
0902 01:59:19.445103 (+ 53002us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.597915 (+152812us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.597957 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.603813 (+ 5856us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:19.603859 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.603951 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.604074 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56117,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":165,"thread_start_us":89349,"threads_started":1}
I20250902 01:59:19.560760 1024 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.565114 1292 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.629478 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1942) took 3078 ms. Trace:
W20250902 01:59:19.640442 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 93386.2us
I20250902 01:59:19.639631 30892 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:19.625713 1515 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250902 01:59:19.651868 1421 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.003s sys 0.013s
I20250902 01:59:19.653256 1024 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.692875 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38110 (request call id 21) took 3987 ms. Trace:
I20250902 01:59:19.693272 31074 rpcz_store.cc:276] 0902 01:59:15.704933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:15.708277 (+ 3344us) service_pool.cc:225] Handling call
0902 01:59:19.692557 (+3984280us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b5400257e54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212a813 00007fc6070901ba 00007fc60072a1c4 00007fc60072a152 00007fc60072a081 00007fc600729ffc 00007fc6007292b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.692673 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4373319040}
W20250902 01:59:19.655756 1547 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.015s sys 0.039s
I20250902 01:59:19.656190 1510 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:19.658612 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1934) took 1763 ms. Trace:
I20250902 01:59:19.691047 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 570) took 1157 ms. Trace:
W20250902 01:59:19.659600 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 61248.9us
I20250902 01:59:19.660588 1516 consensus_queue.cc:1035] T 744f11902df74dea8914fdd4a992259d P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:19.671857 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.673337 1448 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250902 01:59:19.673390 31039 connection.cc:232] RPC call timeout handler was delayed by -1.2126839769999833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20250902 01:59:19.674291 1467 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.005s sys 0.022s
W20250902 01:59:19.695927 1547 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s user 0.015s sys 0.040s
I20250902 01:59:19.676080 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.678244 1637 consensus_queue.cc:1046] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:59:19.680012 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 568) took 1257 ms. Trace:
I20250902 01:59:19.679919 1252 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.681031 1483 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:19.730952 30926 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20250902 01:59:19.682097 30924 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:19.684142 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.639962 1514 raft_consensus.cc:2802] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:19.688181 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 129482us
W20250902 01:59:19.691128 1421 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.595s user 0.005s sys 0.013s
I20250902 01:59:19.645371 1336 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:19.698736 1467 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.005s sys 0.023s
W20250902 01:59:19.748010 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.750501 1443 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:19.754374 1553 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.633122 30919 rpcz_store.cc:276] 0902 01:59:16.517205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.444469 (+927264us) service_pool.cc:225] Handling call
0902 01:59:17.464900 (+ 20431us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:17.465158 (+ 258us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.465181 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:17.470595 (+ 5414us) log.cc:844] Serialized 26 byte log entry
0902 01:59:17.991845 (+521250us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:18.386613 (+394768us) log.cc:844] Serialized 14 byte log entry
0902 01:59:18.500941 (+114328us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.501030 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.365365 (+864335us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:19.551163 (+185798us) raft_consensus.cc:1683] finished
0902 01:59:19.556106 (+ 4943us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.595952 (+ 39846us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122702,"prepare.queue_time_us":300032,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":23,"raft.queue_time_us":276427,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13134,"spinlock_wait_cycles":30521088,"thread_start_us":785989,"threads_started":2,"wal-append.queue_time_us":785243}
I20250902 01:59:19.699223 31073 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:19.751399 1498 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.701332 1510 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.703142 30922 rpcz_store.cc:276] 0902 01:59:18.422448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.492538 (+ 70090us) service_pool.cc:225] Handling call
0902 01:59:18.492867 (+ 329us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.493076 (+ 209us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:18.517880 (+ 24804us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.426183 (+908303us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b180044afa0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d742da 00007fc60213a9da
0902 01:59:19.624908 (+198725us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.630644 (+ 5736us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:19.641292 (+ 10648us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.649185 (+ 7893us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.679848 (+ 30663us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4806,"prepare.queue_time_us":24646,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":11228,"raft.queue_time_us":36224,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5650,"spinlock_wait_cycles":12817280,"thread_start_us":138777,"threads_started":1,"wal-append.queue_time_us":1158118}
I20250902 01:59:19.696316 31000 rpcz_store.cc:276] 0902 01:59:17.895529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.392215 (+1496686us) service_pool.cc:225] Handling call
0902 01:59:19.414980 (+ 22765us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.415684 (+ 704us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.415718 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.419323 (+ 3605us) log.cc:844] Serialized 564 byte log entry
0902 01:59:19.446526 (+ 27203us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:19.446586 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.446662 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.658289 (+211627us) raft_consensus.cc:1683] finished
0902 01:59:19.658308 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.658521 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":26504,"threads_started":1,"wal-append.queue_time_us":205556}
W20250902 01:59:19.755481 1630 log.cc:927] Time spent T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.262s user 0.003s sys 0.000s
W20250902 01:59:19.768850 30994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53772 (request call id 2005) took 1407 ms (client timeout 1781 ms). Trace:
I20250902 01:59:19.756155 30919 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:59:19.770296 30919 raft_consensus.cc:604] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.731278 30926 pending_rounds.cc:85] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Aborting all ops after (but not including) 5
I20250902 01:59:19.771636 30926 pending_rounds.cc:107] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250902 01:59:19.758132 1252 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:19.772133 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.736143 1514 raft_consensus.cc:695] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:19.696780 30921 rpcz_store.cc:276] 0902 01:59:18.493589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.363460 (+869871us) service_pool.cc:225] Handling call
0902 01:59:19.363928 (+ 468us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.364320 (+ 392us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:19.452205 (+ 87885us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.649713 (+197508us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.649850 (+ 137us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.649976 (+ 126us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.650126 (+ 150us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.650833 (+ 707us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10819,"prepare.queue_time_us":313514,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":28625,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":236275,"threads_started":2}
I20250902 01:59:19.773746 1514 consensus_queue.cc:237] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.759248 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.774682 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:19.736603 1515 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:19.775743 1336 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:19.775974 1515 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.756780 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1990) took 2271 ms. Trace:
I20250902 01:59:19.777882 31001 rpcz_store.cc:276] 0902 01:59:17.486399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.197297 (+1710898us) service_pool.cc:225] Handling call
0902 01:59:19.197881 (+ 584us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.370546 (+172665us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.756471 (+385925us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000d75d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.756694 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":395176832}
I20250902 01:59:19.778475 1515 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.736848 1255 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:59:19.778558 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1877) took 1412 ms. Trace:
I20250902 01:59:19.779481 31001 raft_consensus.cc:1273] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:19.764781 31000 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250902 01:59:19.738024 1537 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250902 01:59:19.650417 1500 consensus_queue.cc:237] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:19.740126 1624 log.cc:927] Time spent T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20250902 01:59:19.740630 30998 raft_consensus.cc:1238] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:19.788154 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 487) took 1824 ms. Trace:
I20250902 01:59:19.788539 31000 rpcz_store.cc:276] 0902 01:59:17.963615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.764104 (+1800489us) service_pool.cc:225] Handling call
0902 01:59:19.764509 (+ 405us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.787936 (+ 23427us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.788106 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.742132 30889 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:19.733022 1483 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.746140 1580 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:19.772018 30926 raft_consensus.cc:2887] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:19.779981 31070 rpcz_store.cc:276] 0902 01:59:18.366288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.777807 (+1411519us) service_pool.cc:225] Handling call
0902 01:59:19.778129 (+ 322us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.778306 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.778325 (+ 19us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.778360 (+ 35us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:19.778388 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.778437 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.778515 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.791815 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 488) took 1822 ms. Trace:
I20250902 01:59:19.771844 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:19.792407 31001 rpcz_store.cc:276] 0902 01:59:17.963858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.778675 (+1814817us) service_pool.cc:225] Handling call
0902 01:59:19.779067 (+ 392us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.782009 (+ 2942us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.786135 (+ 4126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:19.793087 30919 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:19.743443 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2000) took 2250 ms. Trace:
I20250902 01:59:19.700484 1448 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:19.746155 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.760893 31729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 674) took 1393 ms. Trace:
I20250902 01:59:19.785848 31071 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.776003 1466 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:19.767539 30920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38796 (request call id 1742) took 3240 ms (client timeout 2299 ms). Trace:
W20250902 01:59:19.769269 30994 rpcz_store.cc:269] 0902 01:59:18.361914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.546861 (+184947us) service_pool.cc:225] Handling call
0902 01:59:19.768657 (+1221796us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b5400127854. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.768796 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1481014656}
I20250902 01:59:19.759467 1498 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.794984 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 576) took 1255 ms. Trace:
W20250902 01:59:19.786487 1623 log.cc:927] Time spent T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.476s user 0.003s sys 0.000s
I20250902 01:59:19.759408 1553 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:19.761677 1292 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.763164 1531 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:19.801836 30892 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:19.795614 30920 rpcz_store.cc:269] 0902 01:59:16.530292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.250482 (+720190us) service_pool.cc:225] Handling call
0902 01:59:19.767182 (+2516700us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54003836d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:19.767344 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":690438912}
I20250902 01:59:19.793828 30999 rpcz_store.cc:276] 0902 01:59:17.492941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.367984 (+1875043us) service_pool.cc:225] Handling call
0902 01:59:19.420725 (+ 52741us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.441395 (+ 20670us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.441443 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.476455 (+ 35012us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.598456 (+122001us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:19.617784 (+ 19328us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.742706 (+124922us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b18000f1c60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff14665a 00007fc608617409 00007fc5ff007af0 00007fc5ff1cb894 00007fc5ff1cb805 00007fc602128020 00007fc60213a6c6 00007fc60214b237 00007fc60214b1b8
0902 01:59:19.742928 (+ 222us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.743009 (+ 81us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.743066 (+ 57us) raft_consensus.cc:1683] finished
0902 01:59:19.743193 (+ 127us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.743362 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5459,"prepare.queue_time_us":19100,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":3349,"raft.queue_time_us":143079,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":6032,"spinlock_wait_cycles":9148160,"thread_start_us":110478,"threads_started":1,"wal-append.queue_time_us":161453}
W20250902 01:59:19.804491 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.751161 1443 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.761201 1510 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:19.797039 1448 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:19.797930 1466 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:19.828896 31070 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:19.823894 31039 leader_election.cc:336] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.511s (SENT)
I20250902 01:59:19.832290 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.765959 1405 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:19.836225 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 131884us
I20250902 01:59:19.794380 31729 rpcz_store.cc:276] 0902 01:59:18.367935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.510937 (+143002us) service_pool.cc:225] Handling call
0902 01:59:18.511235 (+ 298us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:18.566588 (+ 55353us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.566628 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:18.567234 (+ 606us) log.cc:844] Serialized 52 byte log entry
0902 01:59:19.759295 (+1192061us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x7b18003e4160. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171
0902 01:59:19.759618 (+ 323us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.759931 (+ 313us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.760284 (+ 353us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.760443 (+ 159us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.760549 (+ 106us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.760683 (+ 134us) raft_consensus.cc:1683] finished
0902 01:59:19.760701 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.760837 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57453,"prepare.queue_time_us":4097,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":61,"raft.queue_time_us":68981,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":70940,"spinlock_wait_cycles":12311936,"thread_start_us":673450,"threads_started":1,"wal-append.queue_time_us":673751}
I20250902 01:59:19.839104 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250902 01:59:19.841154 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:19.844926 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.845273 31074 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:19.846094 1336 raft_consensus.cc:991] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: : Instructing follower 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:59:19.846851 1467 raft_consensus.cc:1079] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Signalling peer 01f6731ce5044a6a948a5ae04da80e6f to start an election
I20250902 01:59:19.850103 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1874) took 1485 ms. Trace:
I20250902 01:59:19.852031 1499 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:19.850589 31068 rpcz_store.cc:276] 0902 01:59:18.364283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.499728 (+135445us) service_pool.cc:225] Handling call
0902 01:59:18.502693 (+ 2965us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.069918 (+567225us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.069976 (+ 58us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.070644 (+ 668us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.188564 (+117920us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:19.188623 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.190478 (+ 1855us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.781236 (+590758us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:19.849669 (+ 68433us) raft_consensus.cc:1683] finished
0902 01:59:19.849693 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.849927 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":566653,"raft.queue_time_us":573305,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"spinlock_wait_cycles":14095104,"thread_start_us":117392,"threads_started":1,"wal-append.queue_time_us":165826}
I20250902 01:59:19.852538 1499 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:19.853016 1499 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:19.853353 31723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 678) took 1484 ms. Trace:
I20250902 01:59:19.858254 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.859378 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.859880 31072 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.860695 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.861711 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.862665 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.863817 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:19.868033 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2002) took 2329 ms. Trace:
I20250902 01:59:19.869163 1366 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:19.868474 30992 rpcz_store.cc:276] 0902 01:59:17.538325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.385618 (+1847293us) service_pool.cc:225] Handling call
0902 01:59:19.428571 (+ 42953us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.554540 (+125969us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.554567 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.583089 (+ 28522us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.730732 (+147643us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:19.751655 (+ 20923us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.751875 (+ 220us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.751938 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.867790 (+115852us) raft_consensus.cc:1683] finished
0902 01:59:19.867827 (+ 37us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.867962 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50317,"prepare.queue_time_us":20427,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":88354,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":15684,"thread_start_us":116369,"threads_started":1,"wal-append.queue_time_us":216923}
W20250902 01:59:19.871037 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 47197.4us
I20250902 01:59:19.871440 1498 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:59:19.872500 1513 raft_consensus.cc:1062] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: attempting to promote NON_VOTER 01f6731ce5044a6a948a5ae04da80e6f to VOTER
W20250902 01:59:19.875232 1644 log.cc:927] Time spent T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20250902 01:59:19.887634 31073 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:19.890364 31072 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:19.894659 30919 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 9.
I20250902 01:59:19.895447 30923 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:19.853785 31723 rpcz_store.cc:276] 0902 01:59:18.368811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.565003 (+196192us) service_pool.cc:225] Handling call
0902 01:59:18.652715 (+ 87712us) raft_consensus.cc:1469] Updating replica for 3 ops
0902 01:59:18.670382 (+ 17667us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.670410 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0902 01:59:19.067761 (+397351us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b2c000a5ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:59:19.069085 (+ 1324us) log.cc:844] Serialized 78 byte log entry
0902 01:59:19.221769 (+152684us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.303443 (+ 81674us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.387922 (+ 84479us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.399765 (+ 11843us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.852886 (+453121us) raft_consensus.cc:1683] finished
0902 01:59:19.852989 (+ 103us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.853177 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85881,"prepare.queue_time_us":109363,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":97147,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":8234,"spinlock_wait_cycles":13863808,"thread_start_us":212976,"threads_started":2,"wal-append.queue_time_us":524743}
I20250902 01:59:19.801728 1413 consensus_queue.cc:1046] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
W20250902 01:59:19.900151 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 64023.6us
I20250902 01:59:19.906152 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.906940 31068 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:19.903522 31067 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:19.907912 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:19.915264 1406 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.916484 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1867) took 1925 ms. Trace:
I20250902 01:59:19.917014 31069 rpcz_store.cc:276] 0902 01:59:17.991252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.045343 (+ 54091us) service_pool.cc:225] Handling call
0902 01:59:18.072268 (+ 26925us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.343834 (+1271566us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.916089 (+572255us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54000f46d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:19.916406 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2743387520}
W20250902 01:59:19.928987 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 186994us
I20250902 01:59:19.928648 31693 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:19.945248 1564 consensus_meta.cc:287] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.583s user 0.012s sys 0.001s
I20250902 01:59:19.960564 31071 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:19.913774 1515 consensus_queue.cc:1046] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250902 01:59:19.875437 1513 consensus_queue.cc:237] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 4, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.943704 1430 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election won for term 9
W20250902 01:59:19.967600 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:19.967934 1430 raft_consensus.cc:695] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:19.968948 1430 consensus_queue.cc:237] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.969780 1527 consensus_queue.cc:1046] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:59:19.973152 1515 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:19.975381 31073 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:19.973601 1515 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:19.972303 1638 raft_consensus.cc:3053] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:19.994573 1638 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:19.995555 1638 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:19.996678 1405 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:19.996912 1638 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:19.797911 30918 rpcz_store.cc:276] 0902 01:59:18.539201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.516274 (+977073us) service_pool.cc:225] Handling call
0902 01:59:19.516678 (+ 404us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.516954 (+ 276us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:19.638578 (+121624us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.794679 (+156101us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.794733 (+ 54us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:19.794776 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.794831 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:19.794939 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":161202,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":4163,"thread_start_us":272853,"threads_started":2}
I20250902 01:59:19.999495 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:19.993283 1607 log.cc:927] Time spent T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.070s user 0.000s sys 0.009s
I20250902 01:59:20.000697 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.000941 1498 consensus_queue.cc:1046] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:20.001565 30999 raft_consensus.cc:2366] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:20.002003 1498 raft_consensus.cc:3053] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:20.002408 1498 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:20.004155 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1926) took 3360 ms. Trace:
W20250902 01:59:20.005095 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 416796us
I20250902 01:59:20.004529 30993 rpcz_store.cc:276] 0902 01:59:16.644117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.205330 (+1561213us) service_pool.cc:225] Handling call
0902 01:59:18.205651 (+ 321us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:18.206019 (+ 368us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.206046 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:18.206557 (+ 511us) log.cc:844] Serialized 26 byte log entry
0902 01:59:18.421131 (+214574us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:18.421179 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:18.421269 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.992616 (+1571347us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:20.003732 (+ 11116us) raft_consensus.cc:1683] finished
0902 01:59:20.003753 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.004003 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":850,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":78766848,"thread_start_us":214159,"threads_started":1,"wal-append.queue_time_us":715897}
I20250902 01:59:20.007746 1613 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:20.003309 1498 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:20.009416 1621 log.cc:927] Time spent T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.922s user 0.004s sys 0.000s
I20250902 01:59:20.009678 31075 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1719) took 2781 ms. Trace:
I20250902 01:59:20.009909 1498 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.010090 31075 rpcz_store.cc:276] 0902 01:59:17.228531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:17.230458 (+ 1927us) service_pool.cc:225] Handling call
0902 01:59:17.230717 (+ 259us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:17.461423 (+230706us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:17.461447 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:17.461503 (+ 56us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:17.461534 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.009297 (+2547763us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400256f54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:20.009355 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.009532 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":529998080}
I20250902 01:59:20.012037 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51240 (request call id 675) took 1643 ms. Trace:
I20250902 01:59:20.012228 1376 consensus_queue.cc:1046] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20250902 01:59:20.013094 1376 consensus_queue.cc:1230] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer 01f6731ce5044a6a948a5ae04da80e6f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250902 01:59:20.013886 1464 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:20.012454 31727 rpcz_store.cc:276] 0902 01:59:18.368170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.511830 (+143660us) service_pool.cc:225] Handling call
0902 01:59:18.537147 (+ 25317us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:18.662443 (+125296us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:18.662481 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:18.668770 (+ 6289us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.086658 (+417888us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.087145 (+ 487us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.293869 (+206724us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.300774 (+ 6905us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:20.008849 (+708075us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:20.011670 (+ 2821us) raft_consensus.cc:1683] finished
0902 01:59:20.011757 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.011870 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":323789,"prepare.queue_time_us":131236,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":17007,"raft.queue_time_us":385739,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":5387,"spinlock_wait_cycles":325928704,"thread_start_us":308171,"threads_started":1,"wal-append.queue_time_us":397104}
I20250902 01:59:20.014418 1464 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.015197 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.015813 31069 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.016395 1464 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:20.017354 1616 log.cc:927] Time spent T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.617s user 0.000s sys 0.001s
I20250902 01:59:20.018397 1375 raft_consensus.cc:3053] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:20.018757 1375 raft_consensus.cc:738] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:20.012070 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 352096us
I20250902 01:59:20.023097 31069 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
W20250902 01:59:20.023830 31692 leader_election.cc:336] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439): Timed out: RequestConsensusVote RPC to 127.29.21.194:46439 timed out after 1.715s (SENT)
I20250902 01:59:20.024286 31692 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:20.026520 1595 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:20.000715 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 578) took 1419 ms. Trace:
I20250902 01:59:20.028703 1586 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:20.027652 30917 rpcz_store.cc:276] 0902 01:59:18.581381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.518739 (+937358us) service_pool.cc:225] Handling call
0902 01:59:19.519166 (+ 427us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.561315 (+ 42149us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.561345 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.561823 (+ 478us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.887348 (+325525us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:19.912686 (+ 25338us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.912864 (+ 178us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:19.912931 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:19.921597 (+ 8666us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b180044b300. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:20.000427 (+ 78830us) raft_consensus.cc:1683] finished
0902 01:59:20.000527 (+ 100us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.000658 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53465,"prepare.queue_time_us":25153,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":73846,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":20782,"spinlock_wait_cycles":14641536,"thread_start_us":325069,"threads_started":1,"wal-append.queue_time_us":416443}
W20250902 01:59:20.029763 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.030623 1443 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:20.036330 1650 log.cc:927] Time spent T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250902 01:59:20.036543 30992 raft_consensus.cc:2374] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 9. Re-sending same reply.
I20250902 01:59:20.037480 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:20.039032 30965 connection.cc:232] RPC call timeout handler was delayed by -1.4812503499998684s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
W20250902 01:59:20.039403 30965 connection.cc:232] RPC call timeout handler was delayed by -1.4146503499998744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
W20250902 01:59:20.039700 30965 connection.cc:232] RPC call timeout handler was delayed by -1.3741503499999226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20250902 01:59:20.040239 30965 connection.cc:232] RPC call timeout handler was delayed by -1.2553503499999579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
W20250902 01:59:20.040577 30965 connection.cc:232] RPC call timeout handler was delayed by -1.2445503499999404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20250902 01:59:20.041404 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.041996 30999 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:20.043119 1484 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:20.043560 1484 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:20.043967 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1999) took 2551 ms. Trace:
I20250902 01:59:20.044312 30996 rpcz_store.cc:276] 0902 01:59:17.492664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.353119 (+1860455us) service_pool.cc:225] Handling call
0902 01:59:19.422368 (+ 69249us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:19.598681 (+176313us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.598719 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:19.614397 (+ 15678us) log.cc:844] Serialized 26 byte log entry
0902 01:59:19.800903 (+186506us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:19.840556 (+ 39653us) log.cc:844] Serialized 14 byte log entry
0902 01:59:20.012503 (+171947us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.012567 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:20.043737 (+ 31170us) raft_consensus.cc:1683] finished
0902 01:59:20.043755 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.043899 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":185584,"prepare.queue_time_us":47621,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":209842,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":23641,"thread_start_us":150767,"threads_started":1,"wal-append.queue_time_us":274861}
I20250902 01:59:20.035465 31068 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:20.050952 31071 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:20.055135 30892 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:20.055579 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 183722us
I20250902 01:59:20.027000 1466 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:19.966068 1564 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.061331 30892 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:20.071316 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.072714 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.088671 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.092469 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.054414 1580 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:20.112957 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.115422 30892 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:20.119441 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.120646 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.110642 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.125186 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:20.125842 31723 raft_consensus.cc:2366] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:20.126235 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:20.126761 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.127581 31729 raft_consensus.cc:2391] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:20.127478 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.128414 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.128795 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:20.129386 31728 raft_consensus.cc:2374] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 9. Re-sending same reply.
I20250902 01:59:20.131963 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:20.132614 31721 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.135962 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.136678 31726 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:20.137904 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.140354 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.141583 1638 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:20.142171 1638 raft_consensus.cc:695] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:20.143275 1638 consensus_queue.cc:237] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.148053 1511 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:20.148706 1511 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:20.152004 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.152736 30992 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:20.153741 1581 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:20.154421 1581 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.156841 1581 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:20.160615 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.161391 31072 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.178534 1328 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:20.179061 1328 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.180986 1328 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:20.181087 31073 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:20.185935 31070 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:20.072258 1601 raft_consensus.cc:2802] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election won for term 4
W20250902 01:59:20.211891 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:20.204365 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 89026.2us
I20250902 01:59:20.207693 30993 raft_consensus.cc:1273] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250902 01:59:20.221766 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.057726 1359 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.239302 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.245105 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.247377 1405 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:20.247745 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.248039 31070 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:20.248399 31000 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:20.249249 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:20.249718 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.250819 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.251402 31001 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.059933 1515 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.253993 1515 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:20.256197 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.247438 1411 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:20.219671 1366 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:20.259575 30994 raft_consensus.cc:2391] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:20.053277 1375 consensus_queue.cc:260] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.257586 1411 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:20.286124 1448 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.290715 1564 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:20.291949 31724 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:20.271719 30992 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:20.292567 1466 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:20.293555 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.294275 30992 raft_consensus.cc:2391] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:20.295007 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.275033 31723 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6"
dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
from {username='slave'} at 127.0.0.1:51234
I20250902 01:59:20.189006 1580 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.295984 1561 raft_consensus.cc:3053] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:20.296398 1561 raft_consensus.cc:738] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:20.297387 1561 consensus_queue.cc:260] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.298133 1580 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:20.298385 1561 raft_consensus.cc:3058] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.299175 1511 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.268502 1366 raft_consensus.cc:695] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:20.278512 1617 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.816s user 0.003s sys 0.007s
I20250902 01:59:20.278872 1359 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:20.282740 1609 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 1.377s user 0.001s sys 0.017s
I20250902 01:59:20.268478 30996 raft_consensus.cc:1273] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20250902 01:59:20.275820 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:20.320660 30891 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.300137 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.321231 31074 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.275940 31075 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250902 01:59:20.304600 31721 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:20.312098 31072 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:20.308266 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:20.313269 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 344495us
I20250902 01:59:20.333770 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.334453 31001 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:20.337764 31070 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:20.338668 31072 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:20.350371 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:20.377997 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.378505 31073 raft_consensus.cc:1238] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:20.377836 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:20.379217 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 65134.1us
I20250902 01:59:20.335230 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.380827 31000 raft_consensus.cc:1273] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250902 01:59:20.382555 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.384490 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.387540 30994 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
W20250902 01:59:20.394009 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 367907us
W20250902 01:59:20.385478 1531 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.000s
I20250902 01:59:20.395965 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:20.266362 1499 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.398648 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 1993) took 2911 ms. Trace:
I20250902 01:59:20.398844 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:20.432094 30998 rpcz_store.cc:276] 0902 01:59:17.487396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.303331 (+1815935us) service_pool.cc:225] Handling call
0902 01:59:19.308617 (+ 5286us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.791123 (+482506us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.398448 (+607325us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b54002ae9d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:20.398600 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":904728320}
I20250902 01:59:20.402570 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.404811 30992 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:20.390609 1475 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:20.434685 31070 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:20.442062 30921 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:20.445518 31067 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:20.447057 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 47091.5us
I20250902 01:59:20.395310 1366 consensus_queue.cc:237] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.473639 30993 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:20.475168 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 72938.2us
I20250902 01:59:20.480505 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.497608 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:20.501335 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 57161.7us
I20250902 01:59:20.444937 1553 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:20.486347 1483 consensus_meta.cc:287] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.005s
I20250902 01:59:20.503482 31000 raft_consensus.cc:1273] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250902 01:59:20.523728 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.524868 30967 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:20.527066 30999 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:20.528646 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 362589us
W20250902 01:59:20.511081 1406 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.595s user 0.005s sys 0.006s
I20250902 01:59:20.499799 1448 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:20.503345 1430 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.016s sys 0.059s
I20250902 01:59:20.531117 31072 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:20.535779 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.541931 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.544234 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 585) took 1128 ms. Trace:
I20250902 01:59:20.546936 30923 rpcz_store.cc:276] 0902 01:59:19.409036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.635453 (+226417us) service_pool.cc:225] Handling call
0902 01:59:19.674982 (+ 39529us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.895937 (+220955us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.537527 (+641590us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540026fcd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:20.537722 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":506443648}
I20250902 01:59:20.499053 1484 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.501394 1475 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:20.557300 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.515720 1553 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.559051 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.565948 31074 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
W20250902 01:59:20.549636 1566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.048s user 0.017s sys 0.008s
W20250902 01:59:20.545539 1430 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s user 0.016s sys 0.060s
I20250902 01:59:20.517239 1483 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.582813 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.584878 31727 raft_consensus.cc:1238] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:20.579826 1656 log.cc:927] Time spent T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20250902 01:59:20.606554 30997 raft_consensus.cc:2408] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250902 01:59:20.608462 1530 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:20.608932 1483 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:20.581350 1566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.079s user 0.017s sys 0.013s
I20250902 01:59:20.584815 31723 raft_consensus.cc:474] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Not starting forced leader election -- already a leader
I20250902 01:59:20.610212 1527 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:59:20.602706 1411 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:20.614275 1655 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250902 01:59:20.614718 1411 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:20.593583 30925 raft_consensus.cc:3053] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:20.615789 30925 raft_consensus.cc:738] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:20.605602 31070 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:20.608875 1530 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.616844 30925 consensus_queue.cc:260] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.618419 30925 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:20.599807 1433 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20250902 01:59:20.619645 1596 raft_consensus.cc:2802] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:20.620316 1596 raft_consensus.cc:695] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:20.621487 1596 consensus_queue.cc:237] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.622037 1530 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:20.624329 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.635900 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:20.637807 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 43722us
W20250902 01:59:20.644681 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 65017.9us
I20250902 01:59:20.647994 1664 consensus_queue.cc:1035] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20250902 01:59:20.656458 1660 log.cc:927] Time spent T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250902 01:59:20.659888 1499 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:20.662686 30965 leader_election.cc:336] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.299s (SENT)
I20250902 01:59:20.664073 30917 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:20.668068 1484 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:20.585217 1638 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20250902 01:59:20.671466 1553 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:20.577720 1511 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:20.556515 1580 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
W20250902 01:59:20.677125 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.682997 1566 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:20.683930 1566 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:20.684285 1566 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:20.685170 1566 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.686559 1566 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.688181 1375 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.691730 1433 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:20.692166 1537 consensus_queue.cc:1046] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:20.692205 1433 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:20.693235 1537 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:20.693811 1430 raft_consensus.cc:3053] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:20.694311 1430 raft_consensus.cc:738] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:20.693714 1433 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.694988 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.695122 1430 consensus_queue.cc:260] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.696295 1430 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:20.698087 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 51631.8us
I20250902 01:59:20.698802 31730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 977) took 2657 ms. Trace:
I20250902 01:59:20.702638 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.703373 30921 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:20.695396 1433 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:20.708021 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1875) took 2343 ms. Trace:
I20250902 01:59:20.711535 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.717525 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.718164 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:20.718716 31038 leader_election.cc:400] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:20.783279 31038 leader_election.cc:403] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250902 01:59:20.720409 31722 raft_consensus.cc:1238] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:20.722687 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:38794 (request call id 1132) took 1303 ms. Trace:
I20250902 01:59:20.724817 1569 raft_consensus.cc:3053] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Stepping down as leader of term 6
W20250902 01:59:20.726653 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.784135 30924 rpcz_store.cc:276] 0902 01:59:19.419184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.735065 (+315881us) service_pool.cc:225] Handling call
0902 01:59:20.722107 (+987042us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54002a12d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b4cd0 00007fc602138eea 00007fc60709160e 00007fc60072ecc7 00007fc60072ec52 00007fc60072eb81 00007fc60072eafc 00007fc60072ddb2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:20.722585 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1166897920}
W20250902 01:59:20.727731 1417 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.640s user 0.029s sys 0.031s
W20250902 01:59:20.731900 30997 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 482) took 4331 ms (client timeout 2137 ms). Trace:
W20250902 01:59:20.767673 1601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.044s sys 0.055s
I20250902 01:59:20.743038 1579 raft_consensus.cc:3053] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:20.748092 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.750387 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:20.751402 1475 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:20.699134 31730 rpcz_store.cc:276] 0902 01:59:18.041949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.042329 (+ 380us) service_pool.cc:225] Handling call
0902 01:59:18.048611 (+ 6282us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:18.109634 (+ 61023us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:19.219510 (+1109876us) log.cc:844] Serialized 14 byte log entry
0902 01:59:19.435968 (+216458us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.447882 (+ 11914us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:19.459734 (+ 11852us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.698593 (+1238859us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b54004114d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:20.698635 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.698757 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137551,"prepare.queue_time_us":70152,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":132194,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":5035,"spinlock_wait_cycles":139869568,"thread_start_us":1143710,"threads_started":2,"wal-append.queue_time_us":389447,"wal-append.run_cpu_time_us":1560,"wal-append.run_wall_time_us":1043955}
I20250902 01:59:20.754160 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.695607 31070 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:20.755126 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.791018 31067 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:20.791045 30917 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:20.756100 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 486) took 2792 ms. Trace:
I20250902 01:59:20.757139 1366 consensus_queue.cc:1046] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:20.762902 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.765898 29783 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250902 01:59:20.777706 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 115088us
I20250902 01:59:20.765214 31693 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:20.769938 31066 rpcz_store.cc:276] 0902 01:59:18.364546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.314929 (+950383us) service_pool.cc:225] Handling call
0902 01:59:19.315376 (+ 447us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:19.327020 (+ 11644us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:19.327054 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:19.327108 (+ 54us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:19.327143 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.707768 (+1380625us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540024c7d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:20.707820 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.707962 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26312448}
I20250902 01:59:20.733816 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.787091 30920 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:20.785701 30997 rpcz_store.cc:269] 0902 01:59:16.400904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:16.432110 (+ 31206us) service_pool.cc:225] Handling call
0902 01:59:20.731683 (+4299573us) spinlock_profiling.cc:243] Waited 4.06 s on lock 0x7b54002dafd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:20.731830 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9328465664}
W20250902 01:59:20.788219 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 69680.2us
I20250902 01:59:20.793606 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.797485 31069 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250902 01:59:20.796237 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:20.802345 30965 leader_election.cc:336] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.860s (SENT)
W20250902 01:59:20.803345 30965 leader_election.cc:336] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.010s (SENT)
I20250902 01:59:20.803680 30965 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:20.804347 1406 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:20.786250 1569 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:20.806493 30965 leader_election.cc:336] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.066s (SENT)
I20250902 01:59:20.806342 1569 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.806897 30965 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:20.807574 1569 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:20.792833 30995 rpcz_store.cc:276] 0902 01:59:17.963284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:19.582536 (+1619252us) service_pool.cc:225] Handling call
0902 01:59:19.595530 (+ 12994us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:20.011212 (+415682us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:20.011240 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:20.011878 (+ 638us) log.cc:844] Serialized 52 byte log entry
0902 01:59:20.386354 (+374476us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:20.755624 (+369270us) log.cc:844] Serialized 14 byte log entry
0902 01:59:20.755829 (+ 205us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.755890 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:20.755926 (+ 36us) raft_consensus.cc:1683] finished
0902 01:59:20.755936 (+ 10us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:20.756033 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":389318,"prepare.queue_time_us":373000,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":387805,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":4900,"thread_start_us":733210,"threads_started":2,"wal-append.queue_time_us":588206}
I20250902 01:59:20.807652 1530 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:20.811825 30918 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:20.812754 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.813843 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.816118 30926 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:20.816426 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.816969 30920 raft_consensus.cc:2466] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:20.817589 30919 raft_consensus.cc:2391] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:20.817879 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.818336 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.787715 1601 raft_consensus.cc:695] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:20.824349 1595 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:20.824854 1595 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:20.824651 1601 consensus_queue.cc:237] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.825340 1595 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:20.828764 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.830258 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.834228 1366 raft_consensus.cc:3053] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:20.834619 1366 raft_consensus.cc:738] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:20.787515 1417 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.698s user 0.030s sys 0.031s
I20250902 01:59:20.815207 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:20.816877 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.838533 31693 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:20.838073 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.818421 30920 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:20.847810 1483 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:20.889052 30923 raft_consensus.cc:2466] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:20.890959 31038 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:20.892889 30919 raft_consensus.cc:2408] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20250902 01:59:20.893357 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.896726 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.897871 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.900182 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:20.900722 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 58581.3us
I20250902 01:59:20.902096 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:20.848260 1664 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:20.905311 30926 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:20.906260 30920 raft_consensus.cc:2374] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5. Re-sending same reply.
I20250902 01:59:20.911837 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.912967 30924 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:20.890667 1483 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:20.890275 1406 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:20.890298 1530 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:20.926966 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.929004 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.850946 1553 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20250902 01:59:20.934341 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 169940us
W20250902 01:59:20.930269 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 123832us
I20250902 01:59:20.789366 1411 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439) with higher term. Message: Invalid argument: T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:20.936950 30917 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:20.937913 30917 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:20.938517 30921 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250902 01:59:20.789247 1579 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:20.836701 1662 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250902 01:59:20.932606 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:20.823535 1515 consensus_meta.cc:287] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.569s user 0.010s sys 0.000s
I20250902 01:59:20.933574 30926 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:20.948886 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.948527 30820 catalog_manager.cc:5582] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 4 to 6, leader changed from 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 6 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250902 01:59:20.950353 30920 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20250902 01:59:20.940016 1561 consensus_meta.cc:287] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.641s user 0.011s sys 0.000s
W20250902 01:59:20.936404 31695 leader_election.cc:336] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.581s (SENT)
I20250902 01:59:20.951679 31693 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:20.950871 30820 catalog_manager.cc:5582] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:20.942060 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.953809 31695 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:20.956365 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:20.955580 30823 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44296 (request call id 352) took 21219 ms (client timeout 15000 ms). Trace:
I20250902 01:59:20.942906 1412 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader election won for term 9
I20250902 01:59:20.922166 1359 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:20.835423 1366 consensus_queue.cc:260] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.927538 1483 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:20.951506 30920 raft_consensus.cc:1273] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20250902 01:59:20.936182 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:20.952497 30917 raft_consensus.cc:1273] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:20.966113 30923 raft_consensus.cc:2374] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5. Re-sending same reply.
I20250902 01:59:20.960459 31074 raft_consensus.cc:1273] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:20.957862 30924 raft_consensus.cc:2374] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:20.958913 1484 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election won for term 4
I20250902 01:59:20.962700 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.920001 1664 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.931244 31070 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:20.932976 1406 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:20.975244 30892 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:20.975941 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:20.980085 31001 raft_consensus.cc:2408] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:59:20.963186 1359 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:20.990347 1595 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.951664 1530 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:20.993947 31692 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:20.981325 1651 raft_consensus.cc:2802] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:20.981590 1363 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:21.004706 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.005574 30922 raft_consensus.cc:2391] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:59:20.959168 30823 rpcz_store.cc:269] 0902 01:58:59.736626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:58:59.737116 (+ 490us) service_pool.cc:225] Handling call
0902 01:59:20.955319 (+21218203us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0902 01:59:12.321926 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0902 01:59:12.372737 (+ 50811us) write_op.cc:432] Acquiring schema lock in shared mode
0902 01:59:12.372791 (+ 54us) write_op.cc:435] Acquired schema lock
0902 01:59:12.372805 (+ 14us) tablet.cc:660] Decoding operations
0902 01:59:12.374086 (+ 1281us) write_op.cc:620] Acquiring the partition lock for write op
0902 01:59:12.374157 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0902 01:59:12.374186 (+ 29us) tablet.cc:683] Acquiring locks for 14 operations
0902 01:59:12.375322 (+ 1136us) tablet.cc:699] Row locks acquired
0902 01:59:12.375342 (+ 20us) write_op.cc:260] PREPARE: finished
0902 01:59:12.375507 (+ 165us) write_op.cc:270] Start()
0902 01:59:12.426611 (+ 51104us) write_op.cc:276] Timestamp: P: 1756778352375442 usec, L: 0
0902 01:59:12.428102 (+ 1491us) op_driver.cc:348] REPLICATION: starting
0902 01:59:12.496796 (+ 68694us) log.cc:844] Serialized 5629 byte log entry
0902 01:59:12.794769 (+297973us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:13.366026 (+571257us) op_driver.cc:464] REPLICATION: finished
0902 01:59:14.128617 (+762591us) write_op.cc:301] APPLY: starting
0902 01:59:14.149940 (+ 21323us) tablet.cc:1365] starting BulkCheckPresence
0902 01:59:14.282869 (+132929us) tablet.cc:1368] finished BulkCheckPresence
0902 01:59:14.294250 (+ 11381us) tablet.cc:1370] starting ApplyRowOperation cycle
0902 01:59:14.907679 (+613429us) tablet.cc:1381] finished ApplyRowOperation cycle
0902 01:59:14.907934 (+ 255us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0902 01:59:14.907977 (+ 43us) write_op.cc:312] APPLY: finished
0902 01:59:14.908314 (+ 337us) log.cc:844] Serialized 101 byte log entry
0902 01:59:16.304547 (+1396233us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:59:16.334112 (+ 29565us) write_op.cc:454] Released schema lock
0902 01:59:16.346077 (+ 11965us) write_op.cc:341] FINISH: Updating metrics
0902 01:59:16.358647 (+ 12570us) write_op.cc:489] Releasing partition, row and schema locks
0902 01:59:16.358679 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1255099,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":693795,"apply.run_cpu_time_us":113381,"apply.run_wall_time_us":2260125,"num_ops":14,"prepare.queue_time_us":553985,"prepare.run_cpu_time_us":17330,"prepare.run_wall_time_us":400611,"raft.queue_time_us":507696,"raft.run_cpu_time_us":23088,"raft.run_wall_time_us":494197,"replication_time_us":919917,"spinlock_wait_cycles":9606016,"thread_start_us":2469106,"threads_started":5,"wal-append.queue_time_us":1686740,"wal-append.run_cpu_time_us":51376,"wal-append.run_wall_time_us":3257162}]]}
W20250902 01:59:21.008095 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 78169.1us
I20250902 01:59:20.995277 1359 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:21.009299 1530 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:21.011011 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.011610 30920 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:21.011797 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:20.996456 1366 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.012835 31693 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:21.013797 1530 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:21.014313 1530 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.015729 30820 catalog_manager.cc:5582] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:59:21.017247 30820 catalog_manager.cc:5582] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 4 to 5, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:59:20.975682 1412 raft_consensus.cc:695] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:20.981137 1664 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:20.980639 1484 raft_consensus.cc:695] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:20.982753 1406 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:20.955197 1411 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20250902 01:59:21.016316 1530 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:20.981396 1581 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.021s
I20250902 01:59:21.027207 1328 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:21.028121 1412 consensus_queue.cc:237] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.031683 30820 catalog_manager.cc:5582] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 4, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.057611 1579 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.061167 1579 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.034314 30925 raft_consensus.cc:1273] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20250902 01:59:21.062644 30925 pending_rounds.cc:85] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Aborting all ops after (but not including) 5
I20250902 01:59:21.062914 30925 pending_rounds.cc:107] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250902 01:59:21.063153 30925 raft_consensus.cc:2887] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250902 01:59:21.050395 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:21.063786 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.061892 1359 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.067034 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:21.068106 30965 leader_election.cc:336] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.465s (SENT)
I20250902 01:59:21.068506 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:21.069217 1530 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:21.069855 1530 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:21.070399 1530 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.022879 1566 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:21.069943 1680 log.cc:927] Time spent T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.271s user 0.002s sys 0.002s
I20250902 01:59:21.024637 1475 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:21.023097 1651 raft_consensus.cc:695] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:21.088207 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 569) took 2595 ms. Trace:
I20250902 01:59:21.088647 30925 rpcz_store.cc:276] 0902 01:59:18.493291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:18.497522 (+ 4231us) service_pool.cc:225] Handling call
0902 01:59:18.521028 (+ 23506us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:20.397232 (+1876204us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b5400351ea0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6020bf550 00007fc602132170 00007fc6021331e1 00007fc60212fd4f 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c
0902 01:59:21.074734 (+677502us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.088161 (+ 13427us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102998272}
I20250902 01:59:21.095202 30820 catalog_manager.cc:5582] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 4, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:21.100625 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.080572 1475 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:21.086838 1328 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:21.141232 30820 catalog_manager.cc:5582] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:21.146142 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.109614 1510 consensus_queue.cc:1035] T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20250902 01:59:21.086707 1566 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.149562 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.073879 1363 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
W20250902 01:59:21.136746 1368 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20250902 01:59:21.140229 1406 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.154340 30921 raft_consensus.cc:1238] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250902 01:59:21.155081 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.162622 30917 raft_consensus.cc:1273] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250902 01:59:21.140120 1475 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:21.165387 30918 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:21.175663 1461 consensus_queue.cc:1035] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20250902 01:59:21.176599 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.155279 30820 catalog_manager.cc:5582] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.147368 1514 consensus_queue.cc:1035] T d7f66bcc25964883baf67f48889ec766 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20250902 01:59:21.179603 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.180289 31730 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:21.180677 1530 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:21.180173 30820 catalog_manager.cc:5582] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 4, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: HEALTHY } } }
I20250902 01:59:21.181200 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.181787 31724 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:21.182097 30820 catalog_manager.cc:5582] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 6, leader changed from 975ecd397f5146409baac87e56a302e9 (127.29.21.194) to 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193). New cstate: current_term: 6 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.182597 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:21.183195 31724 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:21.183645 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.183976 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.184319 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:21.184551 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:21.184849 31730 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:21.185366 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.185542 30920 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:21.185956 30917 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:21.186489 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.186776 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.187598 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.188227 31721 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:21.188477 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.188884 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.189726 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:21.190855 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.191844 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.193239 31729 raft_consensus.cc:1273] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250902 01:59:21.194180 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:21.194857 31726 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:21.173568 1406 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.198310 1406 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:21.175096 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.184747 31723 raft_consensus.cc:2408] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20250902 01:59:21.203338 30917 raft_consensus.cc:1238] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:21.205211 31725 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:21.207098 31730 raft_consensus.cc:2408] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20250902 01:59:21.173615 1612 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.143784 1375 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:21.208819 1376 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.226716 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.183485 30820 catalog_manager.cc:5582] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 4, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250902 01:59:21.228765 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.225229 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.214895 30918 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:21.231146 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:21.234905 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 226263us
I20250902 01:59:21.236023 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.236783 31730 raft_consensus.cc:2391] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:21.240700 30820 catalog_manager.cc:5582] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 4 to 5, leader changed from 088424002a894fd5bcc11a06af111445 (127.29.21.195) to 975ecd397f5146409baac87e56a302e9 (127.29.21.194). New cstate: current_term: 5 leader_uuid: "975ecd397f5146409baac87e56a302e9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250902 01:59:21.232792 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:21.247253 30918 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:21.247939 30995 raft_consensus.cc:2391] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 8: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:21.254058 31001 raft_consensus.cc:1238] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [term 6 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:21.260362 31070 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:21.261394 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:21.282796 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.259845 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.284348 31730 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 9.
I20250902 01:59:21.217880 31724 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:21.137470 1484 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.287817 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.291738 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.300161 30918 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.232088 1359 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:21.297958 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.305282 1406 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:21.318498 30923 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:21.314393 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:21.321059 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.321353 31001 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.326560 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 1000) took 1052 ms. Trace:
I20250902 01:59:21.327365 31692 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:21.321815 30998 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:21.286355 1595 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:21.239393 1375 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.312956 1368 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20250902 01:59:21.326957 31728 rpcz_store.cc:276] 0902 01:59:20.274461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:20.276755 (+ 2294us) service_pool.cc:225] Handling call
0902 01:59:20.277332 (+ 577us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:20.277759 (+ 427us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:20.277782 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:20.592333 (+314551us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b2c000a5ff8. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:59:20.604008 (+ 11675us) log.cc:844] Serialized 564 byte log entry
0902 01:59:20.905912 (+301904us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:20.905971 (+ 59us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:20.906070 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:21.306315 (+400245us) raft_consensus.cc:1683] finished
0902 01:59:21.306350 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:21.326494 (+ 20144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67643136,"thread_start_us":301511,"threads_started":1,"wal-append.queue_time_us":459956}
I20250902 01:59:21.246778 1376 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:21.323166 1671 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20250902 01:59:21.336937 30820 catalog_manager.cc:5582] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.226056 1651 consensus_queue.cc:237] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.281735 1612 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.344117 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:21.345664 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1951) took 1086 ms. Trace:
I20250902 01:59:21.366475 30992 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:21.370510 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 459408us
I20250902 01:59:21.371228 30918 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:21.397751 30925 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
W20250902 01:59:21.420469 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 83054.1us
I20250902 01:59:21.372777 31693 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:21.424783 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 52100.5us
W20250902 01:59:21.427680 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.429491 30964 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:21.432828 30820 catalog_manager.cc:5582] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "088424002a894fd5bcc11a06af111445" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.336961 1406 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:21.471005 30820 catalog_manager.cc:5582] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
W20250902 01:59:21.472657 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45924.8us
I20250902 01:59:21.297451 1328 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.369100 1595 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:21.477238 31038 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:21.476074 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.485227 1359 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:21.476689 30820 catalog_manager.cc:5582] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 1 to 3, leader changed from 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193) to 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196). New cstate: current_term: 3 leader_uuid: "01f6731ce5044a6a948a5ae04da80e6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:21.488654 30820 catalog_manager.cc:5582] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4372b4c4c866431faa1bf3e91fd47a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } }
I20250902 01:59:21.485911 1359 raft_consensus.cc:695] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:21.469513 1375 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:21.509946 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 283019us
W20250902 01:59:21.494354 1671 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20250902 01:59:21.517879 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.491415 1566 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:21.527637 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.486572 1595 consensus_queue.cc:1046] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20250902 01:59:21.511487 1466 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:21.482563 1483 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.520354 1375 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:21.490164 1692 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
W20250902 01:59:21.541709 1682 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
W20250902 01:59:21.549988 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:21.559509 30965 leader_election.cc:336] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.185s (SENT)
I20250902 01:59:21.560137 30965 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:21.564678 31722 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:21.548627 1595 consensus_queue.cc:1230] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer 975ecd397f5146409baac87e56a302e9 log is divergent from this leader: its last log entry 6.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250902 01:59:21.539258 1581 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.549119 1375 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:21.555655 1466 raft_consensus.cc:695] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:21.594579 31001 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:21.576740 1596 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:21.580160 1375 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:21.602236 1461 raft_consensus.cc:3053] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:21.580472 1376 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:21.611449 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.559783 1359 consensus_queue.cc:237] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:21.606525 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 52100.7us
I20250902 01:59:21.584542 1581 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.661594 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:21.676059 31066 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:21.681309 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 352162us
I20250902 01:59:21.682277 31726 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:21.686734 30922 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:21.657001 1406 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:21.698056 30965 leader_election.cc:336] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 3.296s (SENT)
I20250902 01:59:21.721422 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:21.750249 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 328989us
I20250902 01:59:21.629686 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.627410 1461 raft_consensus.cc:738] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:21.753774 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.772795 30964 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:21.721105 1665 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.773077 31071 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:21.775372 31001 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:21.786772 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:21.787072 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 89068.3us
I20250902 01:59:21.787102 30964 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:21.784420 1708 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20250902 01:59:21.797087 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:21.806927 30992 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:21.810057 31071 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:21.810555 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.812126 1376 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:21.691879 1466 consensus_queue.cc:237] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.772887 1483 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:21.789250 1684 log.cc:927] Time spent T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20250902 01:59:21.796386 1687 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.794059 1406 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:21.814143 1475 consensus_meta.cc:287] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.626s user 0.010s sys 0.000s
I20250902 01:59:21.814918 1376 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:21.823931 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.841688 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:21.842525 31071 raft_consensus.cc:2374] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7. Re-sending same reply.
I20250902 01:59:21.842871 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.843223 1376 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.845494 1376 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:21.839494 1475 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.849792 31721 raft_consensus.cc:2366] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 8. Current term is 9.
W20250902 01:59:21.848946 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 157316us
I20250902 01:59:21.854852 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.821635 1664 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:21.789775 1665 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:21.826117 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.859957 31692 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:21.861614 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 216932us
I20250902 01:59:21.894773 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.897979 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.898691 30922 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.849308 1475 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:21.910144 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:21.910912 31001 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:21.914330 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.915282 30992 raft_consensus.cc:2408] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20250902 01:59:21.917227 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.924263 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.924444 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:21.835460 1693 consensus_queue.cc:1035] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
W20250902 01:59:21.925477 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:21.926015 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.926146 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:21.927130 31069 raft_consensus.cc:2391] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:21.889194 1699 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:21.894443 1376 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:21.900002 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.834164 1687 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.958293 31727 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250902 01:59:21.917214 1483 raft_consensus.cc:2802] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:21.925279 1697 log.cc:927] Time spent T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20250902 01:59:21.957455 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:21.920296 1581 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:21.964232 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.833839 1461 consensus_queue.cc:260] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.970535 30918 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:21.873764 1664 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.950083 1692 consensus_queue.cc:1035] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.139s
I20250902 01:59:21.990442 30922 raft_consensus.cc:2466] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:22.095042 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.094012 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.106941 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.132198 31075 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.145356 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:22.147478 30965 leader_election.cc:336] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.069s (SENT)
I20250902 01:59:22.151656 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.151832 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.152602 30996 raft_consensus.cc:3058] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:22.152680 31724 raft_consensus.cc:2408] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20250902 01:59:22.000989 1711 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
W20250902 01:59:22.094436 1703 log.cc:927] Time spent T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.199s user 0.000s sys 0.000s
W20250902 01:59:22.076962 1695 log.cc:927] Time spent T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.289s user 0.000s sys 0.001s
W20250902 01:59:22.093621 1710 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.184s user 0.000s sys 0.000s
W20250902 01:59:22.158468 31695 leader_election.cc:336] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.741s (SENT)
I20250902 01:59:22.168648 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:22.107312 1696 log.cc:927] Time spent T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.331s user 0.000s sys 0.001s
W20250902 01:59:22.172708 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 207885us
I20250902 01:59:22.079233 1483 raft_consensus.cc:695] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:22.199119 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.021348 31001 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:22.189225 31695 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.192049 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1970) took 1037 ms. Trace:
W20250902 01:59:22.200240 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 90728.3us
I20250902 01:59:22.201637 30921 rpcz_store.cc:276] 0902 01:59:21.154346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.200162 (+ 45816us) service_pool.cc:225] Handling call
0902 01:59:21.200506 (+ 344us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.200888 (+ 382us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.200916 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.201395 (+ 479us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.492550 (+291155us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:21.492598 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.492678 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.191845 (+699167us) raft_consensus.cc:1683] finished
0902 01:59:22.191880 (+ 35us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.191999 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":290744,"threads_started":1,"wal-append.queue_time_us":457464}
I20250902 01:59:22.203599 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.188630 31727 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:21.977610 1705 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:22.205703 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:21.960049 1376 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:21.950871 1699 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:22.189955 1484 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.000s sys 0.008s
W20250902 01:59:22.221215 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 254768us
W20250902 01:59:22.218084 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.218379 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.218765 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.219373 31725 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:22.219825 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:22.213313 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 54506.3us
I20250902 01:59:22.223013 31724 raft_consensus.cc:2408] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20250902 01:59:22.224987 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 289976us
W20250902 01:59:22.263574 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:22.273079 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 50688.5us
I20250902 01:59:22.274161 30965 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:22.284991 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 512) took 1014 ms. Trace:
I20250902 01:59:22.242771 31001 raft_consensus.cc:2391] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:22.274749 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 52353.9us
I20250902 01:59:22.279325 31693 leader_election.cc:304] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:22.287285 31727 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:22.288025 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.288197 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:22.288501 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:22.288841 30889 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:22.289088 30889 connection.cc:232] RPC call timeout handler was delayed by -1.5391527059999817s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20250902 01:59:22.289474 30889 connection.cc:232] RPC call timeout handler was delayed by -1.3510527059999049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20250902 01:59:22.289809 30889 connection.cc:232] RPC call timeout handler was delayed by -1.1170527059999813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2169s for a response
I20250902 01:59:22.275238 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:22.279495 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 54910.5us
W20250902 01:59:22.276513 1711 log.cc:927] Time spent T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250902 01:59:22.293648 31001 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20250902 01:59:22.294265 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 612) took 1060 ms. Trace:
I20250902 01:59:22.295925 1483 consensus_queue.cc:237] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:22.297528 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 84201.8us
I20250902 01:59:22.286890 1705 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.225051 1586 raft_consensus.cc:2747] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:22.298363 30965 leader_election.cc:304] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:22.273720 1412 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.299242 1412 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:22.299844 1705 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.300710 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 518) took 1004 ms. Trace:
I20250902 01:59:22.301015 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 509) took 1049 ms. Trace:
I20250902 01:59:22.301326 1412 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:22.301467 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 610) took 1078 ms. Trace:
I20250902 01:59:22.294660 30917 rpcz_store.cc:276] 0902 01:59:21.233558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.309286 (+ 75728us) service_pool.cc:225] Handling call
0902 01:59:21.334772 (+ 25486us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.335166 (+ 394us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.335203 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.335699 (+ 496us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.620916 (+285217us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:21.620958 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.621033 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.294074 (+673041us) raft_consensus.cc:1683] finished
0902 01:59:22.294095 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.294210 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1240,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":3811,"thread_start_us":278103,"threads_started":1,"wal-append.queue_time_us":474509}
I20250902 01:59:22.302647 30997 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 516) took 1030 ms. Trace:
I20250902 01:59:22.299860 1406 raft_consensus.cc:2747] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:22.301059 30993 rpcz_store.cc:276] 0902 01:59:21.296677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.339326 (+ 42649us) service_pool.cc:225] Handling call
0902 01:59:21.341495 (+ 2169us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.344165 (+ 2670us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.344188 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.422336 (+ 78148us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.852506 (+430170us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:21.852804 (+ 298us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.852986 (+ 182us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.300531 (+447545us) raft_consensus.cc:1683] finished
0902 01:59:22.300549 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.300660 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2303,"raft.queue_time_us":2654,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":14513,"thread_start_us":341350,"threads_started":1,"wal-append.queue_time_us":818424}
I20250902 01:59:22.301476 30999 rpcz_store.cc:276] 0902 01:59:21.251863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.252170 (+ 307us) service_pool.cc:225] Handling call
0902 01:59:21.252559 (+ 389us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.286522 (+ 33963us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:21.320850 (+ 34328us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.619261 (+298411us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.658056 (+ 38795us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.658221 (+ 165us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:21.658260 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.658344 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.300842 (+642498us) raft_consensus.cc:1683] finished
0902 01:59:22.300861 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.300969 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":141557,"prepare.queue_time_us":39940,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":60340,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":26270,"thread_start_us":263265,"threads_started":1,"wal-append.queue_time_us":958782}
I20250902 01:59:22.301815 30919 rpcz_store.cc:276] 0902 01:59:21.222877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.262401 (+ 39524us) service_pool.cc:225] Handling call
0902 01:59:21.305891 (+ 43490us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:21.328184 (+ 22293us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.328213 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:21.354445 (+ 26232us) log.cc:844] Serialized 52 byte log entry
0902 01:59:21.751440 (+396995us) raft_consensus.cc:1643] Marking committed up to 9
0902 01:59:21.792140 (+ 40700us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.792674 (+ 534us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.792871 (+ 197us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.301283 (+508412us) raft_consensus.cc:1683] finished
0902 01:59:22.301299 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.301420 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7451,"prepare.queue_time_us":34015,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":33,"raft.queue_time_us":22922,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":17254,"thread_start_us":315075,"threads_started":1,"wal-append.queue_time_us":449464}
I20250902 01:59:22.285454 31000 rpcz_store.cc:276] 0902 01:59:21.270395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.287347 (+ 16952us) service_pool.cc:225] Handling call
0902 01:59:21.319914 (+ 32567us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:21.344756 (+ 24842us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:21.766890 (+422134us) log.cc:844] Serialized 14 byte log entry
0902 01:59:22.284658 (+517768us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:22.284710 (+ 52us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:22.284748 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:22.284805 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.284929 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101423,"prepare.queue_time_us":216341,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":2912,"raft.queue_time_us":10597,"thread_start_us":734209,"threads_started":2}
W20250902 01:59:22.309625 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.312551 31039 leader_election.cc:304] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:22.221302 1484 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s user 0.001s sys 0.009s
I20250902 01:59:22.292212 30918 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 9.
I20250902 01:59:22.317226 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.275553 1596 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.319505 31723 raft_consensus.cc:2408] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250902 01:59:22.319646 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.329814 31067 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.317966 31724 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:22.320621 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.330554 31073 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.323387 30889 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:22.324787 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.302973 30997 rpcz_store.cc:276] 0902 01:59:21.271786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.312193 (+ 40407us) service_pool.cc:225] Handling call
0902 01:59:21.339029 (+ 26836us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.487014 (+147985us) raft_consensus.cc:1532] Early marking committed up to index 5
0902 01:59:21.514716 (+ 27702us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.767076 (+252360us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.783434 (+ 16358us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.783699 (+ 265us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:21.787435 (+ 3736us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.787641 (+ 206us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.787697 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.302480 (+514783us) raft_consensus.cc:1683] finished
0902 01:59:22.302498 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.302595 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22104,"prepare.queue_time_us":101515,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":84,"raft.queue_time_us":17949,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":84629,"thread_start_us":383166,"threads_started":2,"wal-append.queue_time_us":320876}
I20250902 01:59:22.320588 1720 raft_consensus.cc:2802] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:22.333302 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.311671 1665 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:22.370085 31695 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.835s (SENT)
W20250902 01:59:22.375238 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 71428.7us
W20250902 01:59:22.377988 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.380235 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250902 01:59:22.386025 30997 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250902 01:59:22.386340 1699 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:22.386888 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.367174 1596 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.388832 1596 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.392320 1735 consensus_queue.cc:1035] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250902 01:59:22.393925 1735 consensus_queue.cc:1046] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:22.394672 1412 raft_consensus.cc:3053] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:22.395042 1412 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.395676 1693 consensus_queue.cc:1046] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:59:22.395896 1412 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.396629 1692 raft_consensus.cc:3053] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:22.396919 1412 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:22.397006 1692 raft_consensus.cc:738] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.397866 1692 consensus_queue.cc:260] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:22.399812 1707 log.cc:927] Time spent T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250902 01:59:22.400017 31039 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.401441 1731 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:22.401877 1612 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:22.402199 1729 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.233s user 0.000s sys 0.000s
I20250902 01:59:22.403167 1687 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:22.403761 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 80481.5us
I20250902 01:59:22.404479 30889 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.405650 30889 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:22.406400 1569 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:22.406967 1591 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:22.407068 1569 consensus_queue.cc:1046] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:22.407594 1376 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.328874 1705 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.084s
I20250902 01:59:22.317783 1716 consensus_queue.cc:1046] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
I20250902 01:59:22.370574 31075 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:22.430589 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 547) took 1248 ms. Trace:
W20250902 01:59:22.433575 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 112708us
I20250902 01:59:22.413532 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1964) took 1260 ms. Trace:
I20250902 01:59:22.435248 31721 rpcz_store.cc:276] 0902 01:59:21.182443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.188681 (+ 6238us) service_pool.cc:225] Handling call
0902 01:59:22.430370 (+1241689us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b54006f0754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:22.430523 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1518422912}
I20250902 01:59:22.424397 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 535) took 1245 ms. Trace:
I20250902 01:59:22.436424 31722 rpcz_store.cc:276] 0902 01:59:21.178896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.181968 (+ 3072us) service_pool.cc:225] Handling call
0902 01:59:21.182235 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:21.565168 (+382933us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:22.424101 (+858933us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540059b854. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:22.424326 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":875172736}
I20250902 01:59:22.438649 30996 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:22.454530 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38810 (request call id 1961) took 1285 ms. Trace:
W20250902 01:59:22.454865 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 84816.1us
I20250902 01:59:22.455098 30925 rpcz_store.cc:276] 0902 01:59:21.147717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.154884 (+ 7167us) service_pool.cc:225] Handling call
0902 01:59:22.433411 (+1278527us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54003bea54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:22.433510 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":379487104}
W20250902 01:59:22.371982 1728 log.cc:927] Time spent T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20250902 01:59:22.372859 1726 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20250902 01:59:22.374004 1714 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
W20250902 01:59:22.372427 1722 log.cc:927] Time spent T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.225s user 0.000s sys 0.000s
W20250902 01:59:22.373400 1698 log.cc:927] Time spent T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20250902 01:59:22.372047 1691 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.843s user 0.002s sys 0.000s
I20250902 01:59:22.436585 30924 rpcz_store.cc:276] 0902 01:59:21.152756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.156165 (+ 3409us) service_pool.cc:225] Handling call
0902 01:59:21.177679 (+ 21514us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.177973 (+ 294us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.177997 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.178414 (+ 417us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.502640 (+324226us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:21.558812 (+ 56172us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.559050 (+ 238us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.559133 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.413331 (+854198us) raft_consensus.cc:1683] finished
0902 01:59:22.413365 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.413480 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":60217,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":1308,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":6840,"spinlock_wait_cycles":10130176,"thread_start_us":323836,"threads_started":1,"wal-append.queue_time_us":519003}
W20250902 01:59:22.411089 1375 consensus_meta.cc:287] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.752s user 0.005s sys 0.004s
W20250902 01:59:22.421953 1713 log.cc:927] Time spent T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.444s user 0.002s sys 0.000s
I20250902 01:59:22.472256 30996 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:22.460268 1511 raft_consensus.cc:2747] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:22.400120 1664 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:22.412431 1692 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:22.489989 1375 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:22.501359 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.502727 1375 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.432475 1731 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:22.504036 31038 connection.cc:232] RPC call timeout handler was delayed by -1.0697323520000737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20250902 01:59:22.378890 1534 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:22.504467 31038 connection.cc:232] RPC call timeout handler was delayed by -1.0436323520000315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1562s for a response
W20250902 01:59:22.504834 31038 connection.cc:232] RPC call timeout handler was delayed by -1.0202323519999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
I20250902 01:59:22.506906 31038 leader_election.cc:400] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:22.508098 31038 leader_election.cc:403] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250902 01:59:22.509238 1732 raft_consensus.cc:2747] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285) with higher term. Message: Invalid argument: T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7.
I20250902 01:59:22.510540 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:22.512097 1534 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:22.512548 1534 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:22.512887 1534 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:22.478509 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.513438 30993 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:22.514621 30995 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 9: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:22.516777 1686 raft_consensus.cc:3053] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:22.517033 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1908) took 1495 ms. Trace:
I20250902 01:59:22.456615 1708 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:22.479972 31693 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:22.378881 1720 raft_consensus.cc:695] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:22.490398 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:22.491164 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.488581 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.493132 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.493400 30997 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250902 01:59:22.494879 1566 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:22.497146 1683 log.cc:927] Time spent T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 1.132s user 0.000s sys 0.002s
I20250902 01:59:22.503510 1412 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:22.555778 1412 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.510739 31038 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:22.517406 31074 rpcz_store.cc:276] 0902 01:59:21.021547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.021702 (+ 155us) service_pool.cc:225] Handling call
0902 01:59:21.026582 (+ 4880us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.230536 (+203954us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.230558 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.246234 (+ 15676us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.673827 (+427593us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:21.673865 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.673924 (+ 59us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.516831 (+842907us) raft_consensus.cc:1683] finished
0902 01:59:22.516850 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.516981 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":314044,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":4756,"thread_start_us":566377,"threads_started":2,"wal-append.queue_time_us":859091}
W20250902 01:59:22.557090 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 46453.5us
I20250902 01:59:22.557760 1412 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.558260 1412 raft_consensus.cc:2747] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:22.559242 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.559514 1720 consensus_queue.cc:237] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.560191 1731 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:22.560668 1731 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:22.561053 1731 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:22.563197 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.563916 30924 raft_consensus.cc:2391] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:22.564786 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.565492 30924 raft_consensus.cc:2391] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.566141 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.566844 30924 raft_consensus.cc:2391] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.567111 31038 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:22.569415 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.570689 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.571727 31000 raft_consensus.cc:2408] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250902 01:59:22.519122 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 148147us
I20250902 01:59:22.367945 1406 consensus_queue.cc:1035] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.097s
I20250902 01:59:22.529274 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.582747 30925 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:22.470213 1553 consensus_queue.cc:1046] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20250902 01:59:22.533138 30996 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:22.534605 30993 raft_consensus.cc:2408] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:22.584455 30993 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.9-6.9]
W20250902 01:59:22.530369 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 50496.3us
I20250902 01:59:22.537247 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.537905 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.586314 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.586884 30919 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:22.587630 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.539912 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2dd8f5f814a7d8b48a74020055750" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.540684 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.541754 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.544854 31001 raft_consensus.cc:1238] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:22.545256 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.545477 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.592840 31038 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:22.593750 30996 raft_consensus.cc:2466] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:22.593976 1687 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:22.594660 1687 raft_consensus.cc:695] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.595284 1732 consensus_queue.cc:1046] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:22.595680 1687 consensus_queue.cc:237] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.596426 1732 consensus_queue.cc:1046] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250902 01:59:22.546082 30967 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:22.546690 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.548414 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.548975 31722 raft_consensus.cc:1273] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20250902 01:59:22.599948 31722 pending_rounds.cc:85] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Aborting all ops after (but not including) 7
I20250902 01:59:22.600253 31722 pending_rounds.cc:107] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20250902 01:59:22.600495 31722 raft_consensus.cc:2887] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:22.551411 1700 raft_consensus.cc:2802] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election won for term 9
I20250902 01:59:22.578302 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.603885 1705 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:22.604023 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 58019us
I20250902 01:59:22.605162 30967 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:22.606598 1716 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:22.607332 1716 consensus_queue.cc:1046] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250902 01:59:22.608168 1375 raft_consensus.cc:3053] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:22.608572 1375 raft_consensus.cc:738] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:22.608788 30967 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:22.609755 1716 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:22.609432 1375 consensus_queue.cc:260] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:22.610435 1716 raft_consensus.cc:695] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:22.610860 1375 raft_consensus.cc:3058] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:22.611485 1716 consensus_queue.cc:237] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.617808 30967 leader_election.cc:304] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:22.618547 30919 raft_consensus.cc:2466] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:22.585376 30921 raft_consensus.cc:2374] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6. Re-sending same reply.
I20250902 01:59:22.618942 31000 raft_consensus.cc:1273] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:22.590729 30997 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:22.621666 31001 raft_consensus.cc:1238] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:22.586143 1411 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.524005 1686 raft_consensus.cc:738] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.645460 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.624013 31727 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:22.624310 1591 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.624567 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.625334 30925 raft_consensus.cc:1238] T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:22.625754 30965 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:22.568717 1566 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:22.628047 1688 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250902 01:59:22.629784 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.630437 31730 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:22.631403 30922 raft_consensus.cc:1273] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20250902 01:59:22.631675 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 44189.7us
I20250902 01:59:22.627194 1731 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.633919 1700 raft_consensus.cc:695] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:22.634125 1553 consensus_queue.cc:1230] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250902 01:59:22.658865 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.640417 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.659798 30925 raft_consensus.cc:2391] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 6: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:22.591312 1412 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:22.632301 31724 raft_consensus.cc:1273] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
W20250902 01:59:22.647729 1561 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.022s sys 0.016s
I20250902 01:59:22.652854 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.663296 30922 raft_consensus.cc:2374] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
I20250902 01:59:22.664271 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.665416 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.653685 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.666739 30923 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:22.654937 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.667698 30917 raft_consensus.cc:2374] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:22.668612 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.618953 1665 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:22.671680 1665 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:22.673480 1665 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:22.657408 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.635244 31722 raft_consensus.cc:1273] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEARNER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250902 01:59:22.659686 30924 raft_consensus.cc:2374] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7. Re-sending same reply.
I20250902 01:59:22.674780 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.675273 30924 raft_consensus.cc:2374] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
I20250902 01:59:22.650317 1411 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.676077 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.676497 1735 raft_consensus.cc:3053] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:22.676839 1735 raft_consensus.cc:738] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.646921 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.677745 1411 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:22.677610 1735 consensus_queue.cc:260] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.661980 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:22.645496 1530 raft_consensus.cc:2747] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:22.667431 31695 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.656672 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.687234 30919 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
I20250902 01:59:22.688246 30919 raft_consensus.cc:1215] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[8.9-8.9] Dedup: 8.9->[]
I20250902 01:59:22.688642 30919 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20250902 01:59:22.688663 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 367622us
I20250902 01:59:22.669955 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.662781 31724 pending_rounds.cc:85] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Aborting all ops after (but not including) 6
I20250902 01:59:22.690469 31724 pending_rounds.cc:107] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20250902 01:59:22.690778 31724 raft_consensus.cc:2887] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250902 01:59:22.666292 1561 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s user 0.022s sys 0.017s
I20250902 01:59:22.653167 1406 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:22.688024 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.662782 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.689762 30964 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:22.679592 1727 log.cc:927] Time spent T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.266s user 0.003s sys 0.003s
I20250902 01:59:21.366552 30994 rpcz_store.cc:276] 0902 01:59:20.259264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:20.401624 (+142360us) service_pool.cc:225] Handling call
0902 01:59:20.437043 (+ 35419us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:20.474247 (+ 37204us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:20.913505 (+439258us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.345079 (+431574us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:21.345159 (+ 80us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:21.345283 (+ 124us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.345397 (+ 114us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:21.345584 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":278233,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":5005,"spinlock_wait_cycles":14210304,"thread_start_us":800224,"threads_started":2}
I20250902 01:59:22.679997 30918 raft_consensus.cc:2374] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6. Re-sending same reply.
W20250902 01:59:22.709683 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 120518us
I20250902 01:59:22.710106 30889 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.715951 30920 raft_consensus.cc:1215] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7] Dedup: 6.7->[]
I20250902 01:59:22.716603 30920 raft_consensus.cc:1238] T 7f650d4e1b5f47febbda93daccba0e4f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20250902 01:59:22.717353 31730 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:22.712025 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 603) took 1943 ms. Trace:
I20250902 01:59:22.729735 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.731045 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:22.732601 31074 raft_consensus.cc:1273] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250902 01:59:22.733388 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:22.734191 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 66849.5us
I20250902 01:59:22.734992 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.735528 31727 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:22.735519 30889 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.736866 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.701066 1664 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.747786 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.737396 31727 raft_consensus.cc:2374] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7. Re-sending same reply.
I20250902 01:59:22.745680 31724 raft_consensus.cc:1273] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250902 01:59:22.728490 1561 raft_consensus.cc:3053] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:22.759436 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:22.759791 31074 raft_consensus.cc:1238] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:22.723519 30926 rpcz_store.cc:276] 0902 01:59:20.768295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:20.948257 (+179962us) service_pool.cc:225] Handling call
0902 01:59:20.960641 (+ 12384us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:20.970948 (+ 10307us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:20.970994 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:20.997585 (+ 26591us) log.cc:844] Serialized 52 byte log entry
0902 01:59:21.365656 (+368071us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:21.472837 (+107181us) log.cc:844] Serialized 14 byte log entry
0902 01:59:21.473084 (+ 247us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.473170 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:22.510496 (+1037326us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b180044b540. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:22.688438 (+177942us) raft_consensus.cc:1683] finished
0902 01:59:22.711726 (+ 23288us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:22.711940 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42448,"prepare.queue_time_us":107477,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":10708,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":18628,"spinlock_wait_cycles":23098752,"thread_start_us":357223,"threads_started":1,"wal-append.queue_time_us":498628}
I20250902 01:59:22.701632 1595 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20250902 01:59:22.769328 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.770129 31724 raft_consensus.cc:2374] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7. Re-sending same reply.
I20250902 01:59:22.770757 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.773401 1716 raft_consensus.cc:2802] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250902 01:59:22.773936 1716 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:22.774422 1716 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:22.777040 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:22.777323 31730 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:22.777747 31723 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:22.745353 1665 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:22.746744 1484 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:22.791136 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 196399us
W20250902 01:59:22.791934 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.760510 1734 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:22.760778 1561 raft_consensus.cc:738] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.827710 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.828552 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.829291 31067 raft_consensus.cc:2374] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7. Re-sending same reply.
I20250902 01:59:22.841166 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.763902 1406 consensus_queue.cc:1035] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20250902 01:59:22.843757 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.845299 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:22.847918 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 57651.7us
I20250902 01:59:22.795496 1734 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:22.760427 1664 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.844635 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.845954 31066 raft_consensus.cc:2374] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7. Re-sending same reply.
I20250902 01:59:22.771201 1534 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.859892 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.868321 31073 raft_consensus.cc:2391] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:22.863441 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.859233 31069 raft_consensus.cc:3053] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Stepping down as leader of term 8
I20250902 01:59:22.871299 31069 raft_consensus.cc:738] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:22.871385 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:22.859094 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 145677us
I20250902 01:59:22.841245 1746 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:22.858805 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:22.874707 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 185022us
W20250902 01:59:22.863883 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 105213us
W20250902 01:59:22.880314 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 132618us
I20250902 01:59:22.908456 30889 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:22.909267 30994 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:22.868201 1734 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:22.909186 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.914999 31000 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:22.916052 30996 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:22.911633 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.912052 1699 consensus_queue.cc:1035] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250902 01:59:22.911726 1530 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:22.912989 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:22.880404 31069 consensus_queue.cc:260] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.914453 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.840687 1692 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20250902 01:59:22.958395 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.907261 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.964604 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:22.916849 30994 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:22.821775 1484 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:22.933534 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.959337 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.918340 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.964313 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:22.958066 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:22.958267 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:22.979172 31069 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:22.981348 30997 raft_consensus.cc:3058] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 9
W20250902 01:59:23.001977 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.003516 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.006918 31072 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:22.858230 1705 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:23.009863 30964 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:22.905740 1741 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:23.015518 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.015516 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.914157 1412 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:22.913699 1566 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:23.028627 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.029285 31073 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:23.031347 30996 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:23.001214 1731 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:23.035699 30965 leader_election.cc:336] T ade302bdd69a4fa7876c90a1fbd87b68 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.594s (SENT)
W20250902 01:59:23.043601 30965 leader_election.cc:336] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.668s (SENT)
I20250902 01:59:23.044121 30965 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.044579 31074 raft_consensus.cc:2466] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:22.709569 1739 consensus_queue.cc:1035] T 5cf39ccba06f4a0998fb41dd68a0702a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.338s
I20250902 01:59:22.857683 1742 consensus_queue.cc:1046] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.191s
I20250902 01:59:23.050307 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.053241 31072 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:22.652649 1591 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:22.788136 1700 consensus_queue.cc:237] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.059674 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 54524.5us
I20250902 01:59:23.036444 1566 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.004137 1483 consensus_queue.cc:1035] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20250902 01:59:23.079042 1746 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.003026 1530 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.135381 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:22.980613 1561 consensus_queue.cc:260] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.143108 31695 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.283699 1534 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.189312 1760 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:59:23.296046 30917 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.005s sys 0.001s
W20250902 01:59:23.338721 30917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.005s sys 0.002s
I20250902 01:59:23.198045 1530 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.301277 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:23.220607 1746 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20250902 01:59:23.298380 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:23.178542 1411 consensus_queue.cc:1046] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20250902 01:59:23.201841 1368 raft_consensus.cc:3053] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:23.290716 1751 consensus_queue.cc:1035] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20250902 01:59:23.194203 1750 log.cc:927] Time spent T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.276s user 0.000s sys 0.000s
I20250902 01:59:23.189749 1765 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:23.191896 1484 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:23.193739 1756 log.cc:927] Time spent T 122e9f7b6c1d4295b7d12445ed0116ed P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20250902 01:59:23.079042 1665 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.165186 31069 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:22.980821 1359 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:23.296279 31724 raft_consensus.cc:1238] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 8. Current term is 9. Ops: []
I20250902 01:59:23.052047 1741 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.288693 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.005s sys 0.002s
I20250902 01:59:22.723730 1686 consensus_queue.cc:260] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.291777 30919 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.002s sys 0.005s
I20250902 01:59:23.230280 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:23.384617 30919 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.654s user 0.004s sys 0.007s
W20250902 01:59:23.282198 31723 consensus_meta.cc:287] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.504s user 0.015s sys 0.004s
I20250902 01:59:23.153306 1566 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.331768 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.034124 1412 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.193138 1744 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.352s user 0.000s sys 0.001s
I20250902 01:59:23.228312 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:23.403134 31722 raft_consensus.cc:2391] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:23.170166 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 126631us
I20250902 01:59:23.225502 31727 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:23.221166 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:23.191468 1752 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
W20250902 01:59:23.375923 1375 consensus_meta.cc:287] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.765s user 0.012s sys 0.000s
W20250902 01:59:23.341434 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 433021us
I20250902 01:59:23.330840 1664 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:23.190804 1743 log.cc:927] Time spent T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.276s user 0.000s sys 0.000s
I20250902 01:59:23.346395 1368 raft_consensus.cc:738] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:23.357985 1761 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:23.358626 1780 consensus_queue.cc:1046] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
W20250902 01:59:23.359716 1755 log.cc:927] Time spent T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
W20250902 01:59:23.360289 1730 log.cc:927] Time spent T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.954s user 0.007s sys 0.012s
W20250902 01:59:23.355602 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 212547us
W20250902 01:59:23.491711 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 115058us
I20250902 01:59:23.385792 31723 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 9.
W20250902 01:59:23.380190 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s user 0.005s sys 0.007s
I20250902 01:59:23.492439 31695 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:23.492674 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 734018us
I20250902 01:59:23.385479 31725 raft_consensus.cc:2466] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
W20250902 01:59:23.411099 31039 leader_election.cc:336] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.562s (SENT)
W20250902 01:59:23.472965 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:23.494120 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.397351 1765 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.472843 31075 raft_consensus.cc:2374] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7. Re-sending same reply.
I20250902 01:59:23.477124 30889 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.441638 1686 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:23.502269 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 91227.9us
I20250902 01:59:23.377562 1466 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:23.497059 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 487284us
I20250902 01:59:23.509680 30964 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:23.517132 30964 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:23.521540 30964 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:23.398681 1665 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:23.532986 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 313191us
I20250902 01:59:23.534863 31038 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.495890 1591 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.540601 30964 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.541011 31038 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.542629 31038 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:23.543072 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:23.538823 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 80732us
I20250902 01:59:23.520524 1534 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:23.520864 1375 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:23.531811 1769 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.554268 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:23.554091 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 149923us
I20250902 01:59:23.529639 1761 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250902 01:59:23.550534 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 58178.1us
I20250902 01:59:23.555563 30889 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:23.580262 30965 leader_election.cc:336] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.713s (SENT)
I20250902 01:59:23.591104 30965 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.579527 31074 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:23.605938 31695 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:23.607378 30997 consensus_meta.cc:287] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.000s
I20250902 01:59:23.609018 30997 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 9.
I20250902 01:59:23.614585 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:23.526800 1759 consensus_queue.cc:1046] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20250902 01:59:23.583943 1368 consensus_queue.cc:260] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:23.617285 1737 log.cc:927] Time spent T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 1.209s user 0.000s sys 0.015s
I20250902 01:59:23.617411 1368 raft_consensus.cc:3058] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:23.618525 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 555) took 1763 ms. Trace:
W20250902 01:59:23.619647 1753 log.cc:927] Time spent T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.702s user 0.007s sys 0.003s
I20250902 01:59:23.618834 31728 rpcz_store.cc:276] 0902 01:59:21.854975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.855090 (+ 115us) service_pool.cc:225] Handling call
0902 01:59:21.894653 (+ 39563us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:21.933398 (+ 38745us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.933427 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:22.150030 (+216603us) log.cc:844] Serialized 52 byte log entry
0902 01:59:22.335671 (+185641us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:22.335721 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:22.335799 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:23.537878 (+1202079us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:23.618273 (+ 80395us) raft_consensus.cc:1683] finished
0902 01:59:23.618293 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:23.618474 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18159,"raft.queue_time_us":92869,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":93826,"spinlock_wait_cycles":39349120,"thread_start_us":114119,"threads_started":1,"wal-append.queue_time_us":242223}
I20250902 01:59:23.619922 1693 raft_consensus.cc:2802] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:23.552032 1591 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:23.578342 1789 raft_consensus.cc:3053] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Stepping down as leader of term 4
W20250902 01:59:23.621119 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.549692 1779 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250902 01:59:23.596354 1761 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:23.550016 1692 raft_consensus.cc:2747] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:23.556521 1732 raft_consensus.cc:2747] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.508966 1765 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.620460 1693 raft_consensus.cc:695] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:23.549906 1781 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:23.581040 1466 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:23.676764 31039 leader_election.cc:336] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.714s (SENT)
I20250902 01:59:23.644068 1376 raft_consensus.cc:3053] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:23.611883 1612 raft_consensus.cc:2747] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250902 01:59:23.680449 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 124975us
I20250902 01:59:23.680197 1761 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:23.679816 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:23.679764 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 65263.1us
I20250902 01:59:23.687933 1651 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:23.584335 1534 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.709693 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 103822us
I20250902 01:59:23.752059 31695 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.723676 1770 raft_consensus.cc:2747] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:23.710640 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 517) took 2421 ms. Trace:
I20250902 01:59:23.736392 31039 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:23.756114 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 81064.9us
I20250902 01:59:23.755290 30998 rpcz_store.cc:276] 0902 01:59:21.289751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.337049 (+ 47298us) service_pool.cc:225] Handling call
0902 01:59:21.352024 (+ 14975us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.822712 (+470688us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.822752 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.839127 (+ 16375us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.999488 (+160361us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:22.000239 (+ 751us) log.cc:844] Serialized 14 byte log entry
0902 01:59:22.090983 (+ 90744us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:23.359719 (+1268736us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:23.644119 (+284400us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b54001aefd4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:23.644182 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:23.710444 (+ 66262us) raft_consensus.cc:1683] finished
0902 01:59:23.710463 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:23.710583 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96382,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":189665,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":8946,"spinlock_wait_cycles":196142336,"thread_start_us":470705,"threads_started":2,"wal-append.queue_time_us":518036}
W20250902 01:59:23.756726 31039 leader_election.cc:336] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.750s (SENT)
W20250902 01:59:23.737651 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 157409us
I20250902 01:59:23.586381 1769 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.745162 30889 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.581261 1776 consensus_queue.cc:1046] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.071s
I20250902 01:59:23.738162 1569 raft_consensus.cc:3053] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Stepping down as leader of term 3
I20250902 01:59:23.749590 30995 raft_consensus.cc:3053] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:23.764765 30995 raft_consensus.cc:738] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:23.650560 1789 raft_consensus.cc:738] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:23.734817 1714 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20250902 01:59:23.760756 30965 leader_election.cc:336] T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.842s (SENT)
I20250902 01:59:23.734432 1740 raft_consensus.cc:2747] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.782351 1794 raft_consensus.cc:2747] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:23.782455 1789 consensus_queue.cc:260] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.783488 1802 raft_consensus.cc:2747] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:23.783744 1789 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:23.784830 1749 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20250902 01:59:23.788291 1791 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250902 01:59:23.752004 1651 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:23.792115 1651 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:23.769136 1790 log.cc:927] Time spent T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250902 01:59:23.770478 1769 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:23.771673 1804 raft_consensus.cc:2747] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:59:23.775444 1716 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.000s user 0.009s sys 0.008s
I20250902 01:59:23.764509 1569 raft_consensus.cc:738] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
W20250902 01:59:23.657503 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 108390us
I20250902 01:59:23.739351 1376 raft_consensus.cc:738] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:23.744038 1763 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.751394 1412 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.791754 1466 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.748952 1566 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.777027 30995 consensus_queue.cc:260] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.758571 1759 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.803419 30999 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1966) took 1259 ms. Trace:
I20250902 01:59:23.820559 30999 rpcz_store.cc:276] 0902 01:59:22.544462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.545642 (+ 1180us) service_pool.cc:225] Handling call
0902 01:59:22.545931 (+ 289us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:22.546347 (+ 416us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:22.546372 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:22.546881 (+ 509us) log.cc:844] Serialized 52 byte log entry
0902 01:59:22.800778 (+253897us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:22.800828 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:22.800906 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:23.803207 (+1002301us) raft_consensus.cc:1683] finished
0902 01:59:23.803228 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:23.803366 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1773,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":253529,"threads_started":1,"wal-append.queue_time_us":1065540}
I20250902 01:59:23.822530 30995 raft_consensus.cc:3058] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:23.823318 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 71346.2us
I20250902 01:59:23.824656 31695 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.809567 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:23.826057 31071 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:23.810482 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 61214.8us
W20250902 01:59:23.810899 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 46581.9us
W20250902 01:59:23.800720 30965 leader_election.cc:336] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.845s (SENT)
I20250902 01:59:23.829020 30965 leader_election.cc:304] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.832000 30892 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:23.808801 1765 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:23.836285 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:23.862650 31695 leader_election.cc:336] T 7f650d4e1b5f47febbda93daccba0e4f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.654s (SENT)
I20250902 01:59:23.863672 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.795531 1375 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.833674 31038 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:23.865326 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.865902 31074 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:23.802176 1741 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:23.809054 1786 log.cc:927] Time spent T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.331s user 0.000s sys 0.001s
I20250902 01:59:23.866393 30889 leader_election.cc:304] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:23.867312 30892 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:23.871233 30967 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:23.809487 1774 log.cc:927] Time spent T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
W20250902 01:59:23.874907 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:23.875329 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 74657.9us
W20250902 01:59:23.875936 30965 leader_election.cc:336] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.857s (SENT)
I20250902 01:59:23.864997 1765 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:23.882220 31071 raft_consensus.cc:2408] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:59:23.860450 1553 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:23.861514 1586 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:23.885447 30965 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:23.886775 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.835494 1664 raft_consensus.cc:2747] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.889940 31727 raft_consensus.cc:1273] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250902 01:59:23.890825 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.833235 1781 raft_consensus.cc:695] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:23.892508 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.892782 31073 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:23.893419 30999 raft_consensus.cc:1273] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 4372b4c4c866431faa1bf3e91fd47a88: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250902 01:59:23.895931 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.881711 31723 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:23.893996 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:23.896659 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.896739 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 662) took 1059 ms. Trace:
I20250902 01:59:23.897159 31071 rpcz_store.cc:276] 0902 01:59:22.828531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.828669 (+ 138us) service_pool.cc:225] Handling call
0902 01:59:23.809415 (+980746us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b58001e7898. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff191251 00007fc5ff191102 00007fc607e8cc39 00007fc607018000 00007fc6070deeab 00007fc6070def31 00007fc60708db50 00007fc60708d434 00007fc60071c0c4 00007fc60071c052
0902 01:59:23.887986 (+ 78571us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2754432}
I20250902 01:59:23.807863 1763 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.898193 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.898854 31073 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:23.899742 31071 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:23.900342 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:23.803391 1716 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.879508 1665 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:23.879467 1368 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:23.908373 30892 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:23.883625 1772 raft_consensus.cc:2762] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20250902 01:59:23.903650 31039 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.913391 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:23.914073 31000 raft_consensus.cc:2408] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250902 01:59:23.914693 30967 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:23.883487 1692 raft_consensus.cc:2762] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250902 01:59:23.909766 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:23.917147 30919 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.006s sys 0.010s
I20250902 01:59:23.918246 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:23.924060 31722 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:23.913193 30999 raft_consensus.cc:2953] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } } }
I20250902 01:59:23.808151 1376 consensus_queue.cc:260] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.887590 1553 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:23.804368 1569 consensus_queue.cc:260] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:23.929119 30891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.006s sys 0.011s
I20250902 01:59:23.909626 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.923316 30994 raft_consensus.cc:1238] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20250902 01:59:23.935817 30891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.007s sys 0.011s
I20250902 01:59:23.928287 30998 raft_consensus.cc:1273] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:23.837026 1759 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.931473 30921 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250902 01:59:23.902020 1807 raft_consensus.cc:2747] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250902 01:59:23.926277 30919 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s user 0.007s sys 0.011s
W20250902 01:59:23.941181 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 65267.9us
W20250902 01:59:23.954885 30965 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.067s (SENT) [suppressed 52 similar messages]
W20250902 01:59:23.955889 30965 leader_election.cc:336] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 2.415s (SENT)
I20250902 01:59:23.956375 30965 leader_election.cc:304] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:23.941391 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:23.912053 1368 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:23.959192 1368 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:23.940418 1566 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:23.964098 1566 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:23.966388 1566 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:23.943814 1375 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.923518 1765 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:23.967801 1765 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:23.968214 1765 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:23.948308 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 44764.1us
I20250902 01:59:23.974776 31039 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:23.975765 31039 leader_election.cc:304] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:23.953102 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:23.977401 30997 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 9: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:23.888439 1804 consensus_queue.cc:1035] T cb3bd5884ec54e9da4138c559675467e P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250902 01:59:23.801633 1693 consensus_queue.cc:237] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.981649 30919 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 633) took 1252 ms. Trace:
I20250902 01:59:23.982101 30919 rpcz_store.cc:276] 0902 01:59:22.729681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.729951 (+ 270us) service_pool.cc:225] Handling call
0902 01:59:22.730217 (+ 266us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:22.730383 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 10
0902 01:59:23.385273 (+654890us) log.cc:844] Serialized 14 byte log entry
0902 01:59:23.981229 (+595956us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:23.981289 (+ 60us) raft_consensus.cc:1643] Marking committed up to 10
0902 01:59:23.981342 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:23.981433 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:23.981578 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":470242,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":90,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":1249885,"threads_started":2}
I20250902 01:59:23.989941 1368 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:23.925535 1803 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:23.939155 1734 consensus_meta.cc:287] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.979s user 0.010s sys 0.000s
I20250902 01:59:23.935168 1553 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:23.998250 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:23.999432 30889 connection.cc:232] RPC call timeout handler was delayed by -1.1429986770000369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
I20250902 01:59:23.935469 1534 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.035120 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:24.045639 30998 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:24.040007 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.040916 30918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
W20250902 01:59:24.041616 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:23.996172 30917 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1991) took 1357 ms. Trace:
I20250902 01:59:24.035219 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:24.036731 31000 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:24.046654 30889 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:24.047120 30918 raft_consensus.cc:2408] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20250902 01:59:24.047619 30917 rpcz_store.cc:276] 0902 01:59:22.638573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.701393 (+ 62820us) service_pool.cc:225] Handling call
0902 01:59:22.701663 (+ 270us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:22.702207 (+ 544us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:22.702248 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:22.730132 (+ 27884us) log.cc:844] Serialized 52 byte log entry
0902 01:59:23.339178 (+609046us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:23.339214 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:23.339276 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:23.800968 (+461692us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b4800080058. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc606ff2c41 00007fc606fe9de7 00007fc606fe9c7a 00007fc6070df06f 00007fc6070ded37 00007fc6070f335c 00007fc6070f3308 00007fc6070f3288 00007fc6070f3243 00007fc6070f24ec 00007fc606070656
0902 01:59:23.954020 (+153052us) raft_consensus.cc:1683] finished
0902 01:59:23.954041 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:23.996103 (+ 42062us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7809,"raft.run_cpu_time_us":18911,"raft.run_wall_time_us":1075815,"spinlock_wait_cycles":3664768,"thread_start_us":570446,"threads_started":1,"wal-append.queue_time_us":610639}
I20250902 01:59:23.950241 1776 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:23.952697 1569 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Advancing to term 4
I20250902 01:59:24.063449 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:24.063869 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:24.063962 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 149375us
W20250902 01:59:24.035332 1797 log.cc:927] Time spent T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
I20250902 01:59:24.037971 1781 consensus_queue.cc:237] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.071478 1759 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.076511 30917 raft_consensus.cc:1238] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:24.080396 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.084038 31722 raft_consensus.cc:1273] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250902 01:59:24.084688 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.086423 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.047966 31001 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
W20250902 01:59:24.088207 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.090878 1735 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:24.124522 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 215559us
I20250902 01:59:24.127087 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.092904 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:24.127978 30926 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 9: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:24.141683 31066 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:24.035609 1792 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:24.036197 1368 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.118239 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:24.118427 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 162558us
W20250902 01:59:24.119525 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 45203.8us
W20250902 01:59:23.950829 1770 consensus_queue.cc:1162] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "088424002a894fd5bcc11a06af111445" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 3 index: 6 } }
I20250902 01:59:24.073731 1807 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.143133 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:24.121287 1776 raft_consensus.cc:2747] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.093487 1534 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.147882 30889 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:24.158270 31695 leader_election.cc:336] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.501s (SENT)
I20250902 01:59:24.172369 30997 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.116691 30923 raft_consensus.cc:1238] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6. Ops: [2.5-5.6]
I20250902 01:59:24.085402 1586 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
W20250902 01:59:24.174412 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.173810 1596 raft_consensus.cc:2747] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250902 01:59:24.143338 1566 consensus_queue.cc:1035] T 45b3012a0ae843acbfd03ed33966d585 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20250902 01:59:24.175377 1735 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:24.180588 1772 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:24.184404 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:24.185994 31727 raft_consensus.cc:1238] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20250902 01:59:24.219874 31722 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 9: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:24.087160 1804 consensus_queue.cc:1035] T 5a3db1aaea8d4a3eba2de840361e0849 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20250902 01:59:24.186966 30917 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250902 01:59:24.191058 30889 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:24.223019 30889 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.223100 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.191905 1788 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:24.223740 30923 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:24.223843 1799 raft_consensus.cc:2747] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.074467 1761 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:24.171808 1359 consensus_meta.cc:287] T 92f16b8a15144fcf8807608bdee71123 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.591s user 0.008s sys 0.000s
I20250902 01:59:24.171157 1807 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:24.150799 1725 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
W20250902 01:59:24.207435 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:24.207859 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 125878us
W20250902 01:59:24.208451 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 50221.8us
I20250902 01:59:24.207139 30995 raft_consensus.cc:1273] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20250902 01:59:24.151163 1792 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.145780 1759 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:24.175403 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 199735us
I20250902 01:59:24.215425 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.194988 1808 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:24.200664 1651 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.039422 1734 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.201773 1716 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 10 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.236218 30995 pending_rounds.cc:85] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Aborting all ops after (but not including) 4
I20250902 01:59:24.293975 30995 pending_rounds.cc:107] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250902 01:59:24.294281 30995 raft_consensus.cc:2887] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:24.294603 30995 pending_rounds.cc:107] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250902 01:59:24.294848 30995 raft_consensus.cc:2887] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250902 01:59:24.236732 31695 leader_election.cc:336] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.506s (SENT)
I20250902 01:59:24.237102 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:24.286769 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 50615.4us
I20250902 01:59:24.287302 31039 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.292137 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.297173 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:24.292950 30997 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:24.221292 1776 raft_consensus.cc:2802] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.284690 1368 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.239189 1807 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.288409 30964 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:24.293128 31001 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:59:24.314582 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.297333 31722 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:24.300829 30889 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:24.301643 30925 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:24.302906 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.303948 31728 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
W20250902 01:59:24.293529 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 166510us
I20250902 01:59:24.296142 31695 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.312567 31692 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.234685 1788 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:24.235584 1772 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.340317 1769 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:24.341198 1837 consensus_queue.cc:1035] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20250902 01:59:24.307190 31039 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.313520 1511 raft_consensus.cc:2802] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election won for term 7
W20250902 01:59:24.342150 1806 log.cc:927] Time spent T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.365s user 0.001s sys 0.000s
W20250902 01:59:24.342526 1816 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20250902 01:59:24.342856 1772 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:24.298148 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 61119.1us
I20250902 01:59:24.342833 1822 consensus_queue.cc:1035] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.122s
I20250902 01:59:24.356122 1822 consensus_queue.cc:1035] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250902 01:59:24.357542 1789 consensus_meta.cc:287] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.573s user 0.008s sys 0.000s
I20250902 01:59:24.358158 1784 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:24.341837 1703 log.cc:927] Time spent T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 1.484s user 0.001s sys 0.002s
I20250902 01:59:24.225476 1735 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.366108 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.366753 31723 raft_consensus.cc:2466] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
W20250902 01:59:24.354737 1823 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20250902 01:59:24.344902 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:24.289248 1586 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.024s
I20250902 01:59:24.299064 1808 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:24.364506 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 127769us
I20250902 01:59:24.372751 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:24.373735 1831 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader election won for term 9
I20250902 01:59:24.334338 1664 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250902 01:59:24.387509 31695 leader_election.cc:336] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.022s (SENT)
I20250902 01:59:24.388049 31695 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.310883 1763 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:24.314011 1725 raft_consensus.cc:2802] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.293462 1759 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:24.359493 1811 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.359s user 0.000s sys 0.001s
I20250902 01:59:24.310667 1776 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:24.389961 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 55678.7us
I20250902 01:59:24.392400 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:24.424209 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:24.425294 30889 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.425971 1566 raft_consensus.cc:2747] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250902 01:59:24.377383 1511 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:24.418509 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 73716.3us
I20250902 01:59:24.382884 1837 raft_consensus.cc:2747] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250902 01:59:24.420872 30892 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:24.430009 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.430629 31695 leader_election.cc:304] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.427373 30889 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.435456 31695 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.435196 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:24.439502 31695 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:24.440183 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 133061us
I20250902 01:59:24.394165 1769 raft_consensus.cc:695] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:24.394989 1831 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:24.442790 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.401937 1725 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:24.440829 31039 leader_election.cc:304] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.419265 1761 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.459738 31039 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.433537 1841 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.415210 1765 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.429697 1776 raft_consensus.cc:3058] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.440821 1804 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.463757 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:24.436484 1763 raft_consensus.cc:2747] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.453954 1651 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.472575 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.476374 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.454216 1837 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:24.475841 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.459251 1725 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.481109 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.464811 1586 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.465121 1841 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:24.479435 30919 raft_consensus.cc:2466] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:24.493811 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:24.494169 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.494093 30926 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.494365 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:24.499032 30921 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:24.495175 30917 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:24.491056 31067 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.499905 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:24.504820 30889 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:24.512377 1820 raft_consensus.cc:2747] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:24.515899 1724 log.cc:927] Time spent T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 2.129s user 0.000s sys 0.012s
W20250902 01:59:24.516181 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.516909 1830 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.517629 1830 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:24.517971 1830 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:24.518307 1830 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:24.519937 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1912) took 2949 ms. Trace:
I20250902 01:59:24.497895 1734 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.520273 31070 rpcz_store.cc:276] 0902 01:59:21.570689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.571353 (+ 664us) service_pool.cc:225] Handling call
0902 01:59:21.586474 (+ 15121us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:21.596499 (+ 10025us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:21.596543 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:21.618762 (+ 22219us) log.cc:844] Serialized 26 byte log entry
0902 01:59:21.814482 (+195720us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:21.814527 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:21.815730 (+ 1203us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:24.464309 (+2648579us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:24.519605 (+ 55296us) raft_consensus.cc:1683] finished
0902 01:59:24.519637 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:24.519889 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10588,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8408,"spinlock_wait_cycles":76769152,"thread_start_us":119148,"threads_started":1,"wal-append.queue_time_us":639621}
I20250902 01:59:24.506242 1804 raft_consensus.cc:2802] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.500725 1511 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.474174 1808 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.523530 1789 raft_consensus.cc:2747] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250902 01:59:24.547606 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 71262us
I20250902 01:59:24.502313 1841 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.524900 31067 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
W20250902 01:59:24.550951 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 88021.7us
W20250902 01:59:24.520869 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 79816.7us
I20250902 01:59:24.477778 1769 consensus_queue.cc:237] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.542313 1804 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:24.555485 30965 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
W20250902 01:59:24.567883 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 108225us
I20250902 01:59:24.545689 1844 raft_consensus.cc:2747] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:24.589473 31039 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:24.558459 1359 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:24.567576 1804 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:24.598796 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 244601us
W20250902 01:59:24.596555 1553 consensus_meta.cc:287] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.532s user 0.007s sys 0.002s
I20250902 01:59:24.598904 31039 leader_election.cc:304] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.510568 1831 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.586287 1566 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:24.612578 31039 leader_election.cc:304] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.592635 30965 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.600661 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:24.600841 31695 leader_election.cc:336] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 1.625s (SENT)
I20250902 01:59:24.615834 30998 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 7
I20250902 01:59:24.589110 1759 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.621264 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.605096 1735 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.628108 1759 consensus_queue.cc:1035] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20250902 01:59:24.629436 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:24.630230 30995 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:24.631325 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:24.631759 30965 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:24.633140 1759 raft_consensus.cc:2802] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:24.636165 30965 leader_election.cc:304] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.613935 1822 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
W20250902 01:59:24.646860 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:24.647480 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.648900 31695 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.605422 1359 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.607100 1553 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.622903 1765 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.659044 30965 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:24.634176 1846 raft_consensus.cc:2747] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.640103 1856 raft_consensus.cc:2747] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.665748 30997 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:24.640718 1759 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:24.645190 1596 raft_consensus.cc:2747] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:24.616686 1566 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.680337 30926 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:24.634737 1820 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20250902 01:59:24.659089 1822 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:24.651342 1734 consensus_queue.cc:1046] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20250902 01:59:24.698619 1860 raft_consensus.cc:2747] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:24.669236 1759 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:24.733201 1799 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:24.734436 1359 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.735874 1566 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.737910 1553 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.738651 1869 raft_consensus.cc:2747] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:24.740175 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:24.740789 30999 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:24.741010 30998 raft_consensus.cc:2466] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:24.741802 30889 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:24.742646 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:24.743873 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:24.744350 31074 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:24.744686 31066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 671) took 1836 ms (client timeout 1506 ms). Trace:
W20250902 01:59:24.745033 31066 rpcz_store.cc:269] 0902 01:59:22.908727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.910133 (+ 1406us) service_pool.cc:225] Handling call
0902 01:59:24.744524 (+1834391us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54000406d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:24.744622 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2656335104}
I20250902 01:59:24.746117 30889 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:24.715211 1826 raft_consensus.cc:2802] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:24.749500 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
W20250902 01:59:24.755100 1862 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20250902 01:59:24.755954 1569 consensus_meta.cc:287] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.638s user 0.010s sys 0.000s
I20250902 01:59:24.756328 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2027) took 2207 ms. Trace:
I20250902 01:59:24.747112 1832 raft_consensus.cc:2747] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:24.756721 30993 rpcz_store.cc:276] 0902 01:59:22.548821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.622051 (+ 73230us) service_pool.cc:225] Handling call
0902 01:59:22.622345 (+ 294us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:23.473447 (+851102us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:23.473552 (+ 105us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:23.473702 (+ 150us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:23.473844 (+ 142us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:24.756087 (+1282243us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54002bd254. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:24.756126 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:24.756276 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5816,"raft.queue_time_us":1786,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":12932,"spinlock_wait_cycles":1927502720}
I20250902 01:59:24.756024 1821 consensus_queue.cc:1035] T 69f27fd54f16465fa39cc86d8d6c07bc P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
W20250902 01:59:24.764076 1767 log.cc:927] Time spent T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 1.244s user 0.000s sys 0.003s
I20250902 01:59:24.741969 1763 raft_consensus.cc:3053] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Stepping down as leader of term 7
W20250902 01:59:24.767812 1851 log.cc:927] Time spent T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.202s user 0.000s sys 0.001s
I20250902 01:59:24.752288 1826 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:24.751827 1566 raft_consensus.cc:2802] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:24.758304 1651 raft_consensus.cc:2802] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:24.727356 1840 consensus_queue.cc:1035] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.182s
W20250902 01:59:24.765646 1854 log.cc:927] Time spent T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
I20250902 01:59:24.788573 1566 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:24.792657 1566 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:24.733793 1799 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.801237 1799 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.801388 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 530) took 1089 ms. Trace:
I20250902 01:59:24.779107 1376 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:24.802215 1725 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.801728 30992 rpcz_store.cc:276] 0902 01:59:23.711861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:23.712078 (+ 217us) service_pool.cc:225] Handling call
0902 01:59:23.739389 (+ 27311us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:23.739635 (+ 246us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:23.929722 (+190087us) log.cc:844] Serialized 14 byte log entry
0902 01:59:24.800988 (+871266us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18004787e0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cb805 00007fc602128020 00007fc60213a6c6 00007fc60214b237
0902 01:59:24.801069 (+ 81us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:24.801114 (+ 45us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:24.801159 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:24.801241 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:24.801339 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5573,"prepare.queue_time_us":71056,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":98573,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":65027,"spinlock_wait_cycles":5098368,"thread_start_us":535290,"threads_started":3,"wal-append.queue_time_us":263541}
I20250902 01:59:24.791921 1651 raft_consensus.cc:695] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:24.800055 1807 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.814131 1776 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.815243 31070 raft_consensus.cc:1238] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:24.784250 1763 raft_consensus.cc:738] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:24.804780 1651 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.816151 1763 consensus_queue.cc:260] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.816633 1804 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.817116 1763 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.810993 1830 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.825071 31074 raft_consensus.cc:2466] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:24.826449 30892 leader_election.cc:304] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f
W20250902 01:59:24.827852 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.787338 1826 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:24.849512 30995 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250902 01:59:24.848939 1725 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.852300 1830 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:24.860960 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1986) took 2223 ms. Trace:
I20250902 01:59:24.847754 1368 raft_consensus.cc:2802] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:24.861317 30922 rpcz_store.cc:276] 0902 01:59:22.637191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.691799 (+ 54608us) service_pool.cc:225] Handling call
0902 01:59:22.692374 (+ 575us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:22.712560 (+ 20186us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:22.712583 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:22.770043 (+ 57460us) log.cc:844] Serialized 564 byte log entry
0902 01:59:23.002674 (+232631us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:23.002714 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:23.003160 (+ 446us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:24.763346 (+1760186us) spinlock_profiling.cc:243] Waited 6.22 ms on lock 0x7b440002304c. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc5ff011c41 00007fc5ff0e02b5 00007fc5ff0e0cd2 00007fc601d80231 00007fc601d7bb5a 00007fc601d6f91e 00007fc601d6ec66 00007fc601d6e888 00007fc601d782c1 00007fc601d78279 00007fc601d78209
0902 01:59:24.857391 (+ 94045us) raft_consensus.cc:1683] finished
0902 01:59:24.857415 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:24.860897 (+ 3482us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14296320,"thread_start_us":180236,"threads_started":1,"wal-append.queue_time_us":589549}
W20250902 01:59:24.878077 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 229276us
I20250902 01:59:24.876307 1368 raft_consensus.cc:695] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:24.885109 1368 consensus_queue.cc:237] T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.889648 1664 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:24.890213 1664 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:24.847355 1841 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:24.890097 1566 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:24.905455 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:24.907799 1461 raft_consensus.cc:3058] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:24.913978 1561 raft_consensus.cc:3058] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:24.915928 1807 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:24.917165 1803 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.871s user 0.011s sys 0.000s
I20250902 01:59:24.917580 1803 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:24.919198 1841 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:24.920840 1759 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:24.922715 1759 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:24.820276 1822 consensus_queue.cc:237] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:24.956269 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.089022 1665 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:25.089511 1804 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.089699 1665 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.093744 1665 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.102043 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:25.103230 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.103827 31066 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:25.112202 30892 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:24.900326 1776 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:25.170874 31728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51234 (request call id 1016) took 1290 ms (client timeout 1637 ms). Trace:
W20250902 01:59:25.180187 31728 rpcz_store.cc:269] 0902 01:59:23.880627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:23.886736 (+ 6109us) service_pool.cc:225] Handling call
0902 01:59:25.170620 (+1283884us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54004d4754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:25.170803 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":873788288}
I20250902 01:59:25.193372 30920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 1968) took 4039 ms. Trace:
I20250902 01:59:25.193876 30920 rpcz_store.cc:276] 0902 01:59:21.153845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.189354 (+ 35509us) service_pool.cc:225] Handling call
0902 01:59:22.715828 (+1526474us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:22.716969 (+ 1141us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.192934 (+2475965us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54002ca6d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:25.193317 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3510200704}
W20250902 01:59:25.205899 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.206790 31068 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:25.208724 1406 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:25.211417 1794 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.211664 1406 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.211920 1794 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:25.212808 31721 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 694) took 2662 ms (client timeout 2194 ms). Trace:
W20250902 01:59:25.213219 31721 rpcz_store.cc:269] 0902 01:59:22.546541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.548200 (+ 1659us) service_pool.cc:225] Handling call
0902 01:59:25.209008 (+2660808us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54006f0754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:25.209124 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2635729152}
I20250902 01:59:25.213536 1406 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.214080 1406 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:25.214489 1406 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:25.217170 1803 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.217690 1803 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.201375 1484 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:25.221230 1484 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.222875 1484 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:25.208985 1534 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.223714 1534 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.225479 1534 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.219391 1803 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.229151 1406 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.238394 31073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.238958 31073 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 3.
I20250902 01:59:25.193490 1665 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:25.241241 1665 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:25.241614 1665 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:25.243057 30892 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:25.247210 1731 raft_consensus.cc:3053] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:25.247987 1731 raft_consensus.cc:738] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:25.249320 1731 consensus_queue.cc:260] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:25.248015 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:25.254942 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.213629 1794 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.273741 31726 raft_consensus.cc:1273] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
W20250902 01:59:25.275972 30921 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.044s sys 0.038s
W20250902 01:59:25.279012 30921 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.044s sys 0.038s
W20250902 01:59:25.282922 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.027s sys 0.064s
W20250902 01:59:25.283267 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.027s sys 0.064s
I20250902 01:59:25.250672 1746 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:25.295357 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.295979 31723 raft_consensus.cc:3058] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:25.295992 31727 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:25.307737 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.309336 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.310050 31727 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:25.308477 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.317003 31723 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:25.242180 1664 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.322204 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.322331 1566 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.292913 1746 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:25.421195 1561 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:25.366263 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.051s sys 0.049s
W20250902 01:59:25.504693 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.052s sys 0.062s
W20250902 01:59:25.351444 30890 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:25.478169 31695 leader_election.cc:336] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531): Timed out: RequestConsensusVote RPC to 127.29.21.195:40531 timed out after 2.845s (SENT)
I20250902 01:59:25.505860 31695 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:25.497184 1701 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.413061 1328 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.345222 1461 raft_consensus.cc:491] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:25.323621 1781 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.018s sys 0.063s
W20250902 01:59:25.507591 1781 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.026s sys 0.075s
W20250902 01:59:25.384349 1511 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.020s sys 0.021s
W20250902 01:59:25.505609 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 846640us
W20250902 01:59:25.508394 1511 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s user 0.020s sys 0.025s
I20250902 01:59:25.508826 30965 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:25.364415 1802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.407s user 0.035s sys 0.067s
W20250902 01:59:25.391731 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.506784 1328 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.518754 30965 leader_election.cc:304] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:25.515640 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.346292 1566 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
W20250902 01:59:25.519449 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.872s user 0.006s sys 0.045s
W20250902 01:59:25.520910 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.873s user 0.007s sys 0.045s
I20250902 01:59:25.506551 1701 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.522038 30964 leader_election.cc:304] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:25.364063 1794 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.532174 30997 raft_consensus.cc:1273] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20250902 01:59:25.526353 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.001s sys 0.021s
W20250902 01:59:25.534065 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.919s user 0.001s sys 0.022s
W20250902 01:59:25.534543 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 919680us
I20250902 01:59:25.504496 1746 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:25.517206 1802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.572s user 0.045s sys 0.074s
I20250902 01:59:25.509302 1601 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:25.544704 1601 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.545154 1781 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:25.545624 1781 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.546523 1601 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.547981 1328 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.526008 1803 consensus_queue.cc:1046] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20250902 01:59:25.533437 1566 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.520692 1759 raft_consensus.cc:2747] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250902 01:59:25.556768 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 78649.9us
W20250902 01:59:25.517669 1831 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.009s sys 0.026s
W20250902 01:59:25.525981 1818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.011s sys 0.013s
I20250902 01:59:25.551394 1566 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:25.570976 31722 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.973s user 0.033s sys 0.088s
W20250902 01:59:25.562315 1831 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s user 0.009s sys 0.026s
I20250902 01:59:25.507490 1461 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:25.572691 31722 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.975s user 0.033s sys 0.088s
I20250902 01:59:25.420922 1406 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.560331 1759 raft_consensus.cc:2747] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250902 01:59:25.568547 1818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.011s sys 0.013s
I20250902 01:59:25.560529 1699 consensus_queue.cc:1046] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.011s
I20250902 01:59:25.535455 1561 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.539897 1794 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:25.583487 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.052s sys 0.068s
W20250902 01:59:25.638448 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.062s sys 0.071s
W20250902 01:59:25.575860 1769 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.003s sys 0.045s
W20250902 01:59:25.640619 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.08259e+06us
I20250902 01:59:25.572302 1872 raft_consensus.cc:2747] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250902 01:59:25.629983 1699 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:25.642230 1769 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.924s user 0.005s sys 0.046s
I20250902 01:59:25.630837 1461 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.650987 1699 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.648366 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:25.655114 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.632591 1664 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:25.639375 1831 raft_consensus.cc:2802] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:25.645973 1826 consensus_meta.cc:287] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.790s user 0.006s sys 0.011s
W20250902 01:59:25.665030 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.665122 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:25.670213 30925 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:25.677078 30967 leader_election.cc:304] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:25.678690 31728 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:25.678695 30967 leader_election.cc:304] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:25.613843 1406 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.647282 1759 raft_consensus.cc:3053] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:25.683485 1826 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.688305 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:25.688565 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 526) took 2733 ms. Trace:
I20250902 01:59:25.688838 30997 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:25.689474 1826 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.690140 1826 raft_consensus.cc:2747] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:25.689043 30996 rpcz_store.cc:276] 0902 01:59:22.954943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.956444 (+ 1501us) service_pool.cc:225] Handling call
0902 01:59:22.956753 (+ 309us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:23.031738 (+ 74985us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.688335 (+2656597us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b54001aead4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:25.688514 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171158272}
I20250902 01:59:25.650243 1781 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.660173 1742 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.671689 1831 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:25.682711 1769 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.708077 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:25.724967 31068 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 672) took 2816 ms (client timeout 1625 ms). Trace:
W20250902 01:59:25.725356 31068 rpcz_store.cc:269] 0902 01:59:22.908967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.909623 (+ 656us) service_pool.cc:225] Handling call
0902 01:59:25.717445 (+2807822us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b54000f32d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:25.724906 (+ 7461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4875805568}
I20250902 01:59:25.726385 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:25.727249 31071 raft_consensus.cc:1273] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Refusing update from remote peer 01f6731ce5044a6a948a5ae04da80e6f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20250902 01:59:25.731633 30925 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 9.
I20250902 01:59:25.694959 1759 raft_consensus.cc:738] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:25.733439 30924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.928s user 0.033s sys 0.026s
W20250902 01:59:25.735095 30924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s user 0.033s sys 0.026s
I20250902 01:59:25.699955 1831 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:25.700511 1742 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.739464 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:25.740367 31068 raft_consensus.cc:2391] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:25.741514 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.739638 1742 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:25.742360 31068 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:25.739668 31693 leader_election.cc:304] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:25.743238 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.743455 1417 raft_consensus.cc:2802] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election won for term 9
I20250902 01:59:25.744019 1417 raft_consensus.cc:695] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:25.744148 31068 raft_consensus.cc:2408] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20250902 01:59:25.745009 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:25.744928 1417 consensus_queue.cc:237] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.745797 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.746078 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.740288 1720 raft_consensus.cc:3053] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Stepping down as leader of term 5
I20250902 01:59:25.747184 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:25.747254 1720 raft_consensus.cc:738] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:25.748097 31068 raft_consensus.cc:2408] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 8 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20250902 01:59:25.748378 1699 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:25.748250 1794 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.748456 1720 consensus_queue.cc:260] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.749676 1720 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:25.751149 31068 raft_consensus.cc:1238] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20250902 01:59:25.728399 1763 consensus_meta.cc:287] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.911s user 0.011s sys 0.000s
I20250902 01:59:25.701951 1781 raft_consensus.cc:2802] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:25.754364 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:25.760607 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:25.781021 31721 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:25.782392 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:25.769279 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.780712 30917 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 9.
W20250902 01:59:25.754400 1665 consensus_meta.cc:287] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.512s user 0.007s sys 0.003s
W20250902 01:59:25.753304 1880 log.cc:927] Time spent T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
W20250902 01:59:25.785300 30889 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.967s user 0.029s sys 0.051s
W20250902 01:59:25.791798 30889 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.974s user 0.029s sys 0.052s
I20250902 01:59:25.787784 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:25.792105 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.0209e+06us
I20250902 01:59:25.789266 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.789701 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:25.793051 31001 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:25.813534 30996 raft_consensus.cc:1238] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:25.792811 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.815624 31725 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:25.812225 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:25.818596 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:25.820087 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.830039 31068 raft_consensus.cc:1238] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:25.830533 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.832365 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.832839 31723 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:25.835958 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.829478 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:25.837252 1793 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.843159 31001 raft_consensus.cc:2408] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 8 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20250902 01:59:25.843480 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:25.844269 31725 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:25.839995 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:25.733505 1769 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: true } }
I20250902 01:59:25.798699 1781 raft_consensus.cc:695] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [term 8 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:25.820465 1794 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:25.841275 1601 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.827637 1765 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:25.841539 1751 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:25.868532 1844 raft_consensus.cc:491] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:25.872458 1601 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.873250 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1977) took 1959 ms. Trace:
I20250902 01:59:25.876487 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2039) took 1799 ms. Trace:
I20250902 01:59:25.865676 1812 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250902 01:59:25.841094 1804 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:25.815008 1665 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.878634 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:25.861407 1561 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.885751 1751 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.885890 30994 rpcz_store.cc:276] 0902 01:59:23.913375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:23.914381 (+ 1006us) service_pool.cc:225] Handling call
0902 01:59:23.920340 (+ 5959us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:23.935890 (+ 15550us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.872973 (+1937083us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b54002bdc54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:25.873188 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6264704}
W20250902 01:59:25.922281 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 43740.1us
I20250902 01:59:25.886493 31000 rpcz_store.cc:276] 0902 01:59:24.077314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.081300 (+ 3986us) service_pool.cc:225] Handling call
0902 01:59:24.081625 (+ 325us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:24.081999 (+ 374us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:24.082035 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:24.082585 (+ 550us) log.cc:844] Serialized 26 byte log entry
0902 01:59:24.735181 (+652596us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b180020cfe0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce8bf 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:25.548630 (+813449us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b180020cfe0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce792 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:25.874698 (+326068us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b180020cfe0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff19c786 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b
0902 01:59:25.875894 (+ 1196us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b180020cfe0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc601d7c55c 00007fc5ff1b177b 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171
0902 01:59:25.876063 (+ 169us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:25.876112 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.876202 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:25.876275 (+ 73us) raft_consensus.cc:1683] finished
0902 01:59:25.876294 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:25.876425 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107449,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":14136,"spinlock_wait_cycles":49086080,"thread_start_us":116336,"threads_started":1,"wal-append.queue_time_us":116314}
I20250902 01:59:25.927003 31723 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:25.861901 1793 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.941895 31695 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:25.943444 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:25.948247 31001 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:25.944752 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:25.882057 1845 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s user 0.016s sys 0.073s
W20250902 01:59:25.950307 1845 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.101s user 0.016s sys 0.073s
W20250902 01:59:25.952032 1882 log.cc:927] Time spent T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20250902 01:59:25.949878 1877 consensus_queue.cc:1046] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.010s
W20250902 01:59:25.955971 1885 log.cc:927] Time spent T 00000000000000000000000000000000 P 7a3020b0c4d54495a77a3aff71003d3b: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20250902 01:59:25.956265 30992 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250902 01:59:25.957145 1781 consensus_queue.cc:237] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.880300 1792 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.945010 31038 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:25.897956 1812 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:25.971895 30996 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:25.892576 1406 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.861187 1759 consensus_queue.cc:260] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.926132 1856 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:25.976459 31038 leader_election.cc:304] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:25.882725 1794 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:25.882611 1765 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.895998 1844 raft_consensus.cc:513] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:25.983793 1802 raft_consensus.cc:2802] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:25.988461 1751 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:25.887578 1651 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.040s user 0.032s sys 0.058s
I20250902 01:59:25.993078 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 537) took 1365 ms. Trace:
I20250902 01:59:25.995779 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 658) took 1485 ms. Trace:
I20250902 01:59:25.889381 1601 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:25.977074 1812 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:25.968863 1328 raft_consensus.cc:2747] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250902 01:59:25.957176 1788 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:25.937026 1804 raft_consensus.cc:513] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.001034 30995 rpcz_store.cc:276] 0902 01:59:24.628142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.632242 (+ 4100us) service_pool.cc:225] Handling call
0902 01:59:24.632570 (+ 328us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:24.850058 (+217488us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.992844 (+1142786us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54002ad0d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:25.993026 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498352896}
I20250902 01:59:25.976747 1849 raft_consensus.cc:491] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.004508 31038 leader_election.cc:304] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 088424002a894fd5bcc11a06af111445; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:26.000717 1651 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s user 0.033s sys 0.058s
W20250902 01:59:26.009162 31074 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.006s sys 0.026s
I20250902 01:59:26.001117 30921 rpcz_store.cc:276] 0902 01:59:24.509983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.515605 (+ 5622us) service_pool.cc:225] Handling call
0902 01:59:24.515918 (+ 313us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:24.540130 (+ 24212us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:24.585042 (+ 44912us) log.cc:844] Serialized 14 byte log entry
0902 01:59:25.284287 (+699245us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:25.311774 (+ 27487us) log.cc:844] Serialized 26 byte log entry
0902 01:59:25.311951 (+ 177us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:25.311980 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.312069 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:25.995605 (+683536us) raft_consensus.cc:1683] finished
0902 01:59:25.995626 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:25.995727 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27093,"prepare.queue_time_us":83019,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":28629,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":16402,"thread_start_us":703224,"threads_started":2,"wal-append.queue_time_us":964677}
I20250902 01:59:26.016505 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:26.015029 31074 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.771s user 0.006s sys 0.026s
I20250902 01:59:26.016220 1751 raft_consensus.cc:2747] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:26.019786 1769 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.021353 1840 consensus_queue.cc:1046] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250902 01:59:26.025709 31075 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 670) took 3117 ms (client timeout 2845 ms). Trace:
W20250902 01:59:26.044226 31075 rpcz_store.cc:269] 0902 01:59:22.908393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:22.908497 (+ 104us) service_pool.cc:225] Handling call
0902 01:59:26.025529 (+3117032us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b5400157054. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:26.025643 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1083997056}
I20250902 01:59:26.028354 1877 consensus_queue.cc:1046] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20250902 01:59:26.030583 1804 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:25.980036 1763 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20250902 01:59:26.001578 1802 raft_consensus.cc:491] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:25.997143 1883 raft_consensus.cc:2747] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:26.011703 1759 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:26.046892 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:26.079337 30995 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
W20250902 01:59:26.080037 31072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1942) took 2087 ms (client timeout 1703 ms). Trace:
W20250902 01:59:26.080451 31072 rpcz_store.cc:269] 0902 01:59:23.993075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.303505 (+310430us) service_pool.cc:225] Handling call
0902 01:59:26.079864 (+1776359us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400201554. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:26.079978 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2745857408}
I20250902 01:59:26.063352 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:26.081388 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:26.081596 31075 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
W20250902 01:59:26.065868 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.013527 1856 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:26.082333 31066 raft_consensus.cc:3053] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:26.074882 1840 raft_consensus.cc:3053] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 LEADER]: Stepping down as leader of term 9
I20250902 01:59:26.075487 1877 raft_consensus.cc:3053] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:26.065721 1802 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:26.077803 30924 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38796 (request call id 1811) took 2038 ms. Trace:
W20250902 01:59:26.078620 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.009562 1788 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:26.074465 1763 raft_consensus.cc:3053] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:26.082799 31066 raft_consensus.cc:738] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:26.115271 1746 consensus_meta.cc:287] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.557s user 0.009s sys 0.000s
I20250902 01:59:26.115984 1701 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.115741 1746 raft_consensus.cc:513] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.118096 1746 leader_election.cc:290] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:26.119076 30920 raft_consensus.cc:2391] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:26.113821 31066 consensus_queue.cc:260] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.015787 1849 raft_consensus.cc:513] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.090361 31070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1767) took 1647 ms. Trace:
I20250902 01:59:26.152796 31070 rpcz_store.cc:276] 0902 01:59:24.442639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.565833 (+123194us) service_pool.cc:225] Handling call
0902 01:59:24.581401 (+ 15568us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:24.815601 (+234200us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.090101 (+1274500us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400020154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:26.090296 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537451392}
I20250902 01:59:26.061384 1687 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:26.039996 1787 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:26.105129 30924 rpcz_store.cc:276] 0902 01:59:24.039315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.063962 (+ 24647us) service_pool.cc:225] Handling call
0902 01:59:24.064249 (+ 287us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:24.119234 (+ 54985us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:24.119258 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:24.804356 (+685098us) spinlock_profiling.cc:243] Waited 998 us on lock 0x7b2c0004d918. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:59:24.804765 (+ 409us) log.cc:844] Serialized 26 byte log entry
0902 01:59:25.735482 (+930717us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:25.737521 (+ 2039us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:25.737626 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:26.077475 (+339849us) raft_consensus.cc:1683] finished
0902 01:59:26.077500 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:26.077738 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54587,"raft.queue_time_us":54921,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":24416,"spinlock_wait_cycles":2295168,"thread_start_us":930335,"threads_started":1,"wal-append.queue_time_us":1257944}
I20250902 01:59:26.148094 31066 raft_consensus.cc:3058] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:26.112402 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:26.084506 31038 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:26.063335 1799 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
W20250902 01:59:26.167212 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 82808us
I20250902 01:59:26.109541 1328 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.066119 1751 consensus_queue.cc:1046] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20250902 01:59:26.092253 1840 raft_consensus.cc:738] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:26.104765 1877 raft_consensus.cc:738] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:26.175110 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:26.175616 1879 log.cc:927] Time spent T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20250902 01:59:26.175897 1761 consensus_queue.cc:1046] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20250902 01:59:26.180935 1849 leader_election.cc:290] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.182009 1793 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:26.159262 1701 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.185657 30919 raft_consensus.cc:1238] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:26.187664 1328 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.160463 1687 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:26.172153 1787 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.172006 1883 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:26.193562 1794 leader_election.cc:290] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:26.173935 1665 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:26.357710 30964 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.010s sys 0.030s
W20250902 01:59:26.358001 30964 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.830s user 0.010s sys 0.030s
W20250902 01:59:26.358330 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 836389us
I20250902 01:59:26.194888 1844 leader_election.cc:290] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.113171 1763 raft_consensus.cc:738] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
W20250902 01:59:26.203915 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.206858 31075 raft_consensus.cc:2466] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
W20250902 01:59:26.217978 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:26.325299 30925 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.010s sys 0.029s
I20250902 01:59:26.191269 1840 consensus_queue.cc:260] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.347648 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:26.352737 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.019s sys 0.024s
I20250902 01:59:26.459966 1765 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.361449 1883 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:26.352077 1687 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
W20250902 01:59:26.406132 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.012s sys 0.035s
W20250902 01:59:26.772927 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.246s user 0.018s sys 0.049s
W20250902 01:59:26.773555 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 1.25487e+06us
I20250902 01:59:26.326191 1804 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.774890 30965 leader_election.cc:304] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:26.321779 1770 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.693077 30964 leader_election.cc:304] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:26.778613 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 85633us
W20250902 01:59:26.458757 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.780447 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:26.781205 31038 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
W20250902 01:59:26.409778 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.699885 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:26.109215 1856 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:26.722527 30925 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.937s user 0.036s sys 0.083s
W20250902 01:59:26.722575 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.164s user 0.019s sys 0.024s
W20250902 01:59:26.788741 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:26.725838 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 401360us
I20250902 01:59:26.728288 31721 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56978 (request call id 574) took 1408 ms. Trace:
I20250902 01:59:26.689853 1651 raft_consensus.cc:491] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:26.732064 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:26.732477 31727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.055s sys 0.109s
W20250902 01:59:26.796904 31727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s user 0.055s sys 0.110s
W20250902 01:59:26.763779 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:26.357033 1701 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.789718 31721 rpcz_store.cc:276] 0902 01:59:25.319473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.515380 (+195907us) service_pool.cc:225] Handling call
0902 01:59:26.728090 (+1212710us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54007413d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:26.728221 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":600234880}
I20250902 01:59:26.729774 30926 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 663) took 2120 ms. Trace:
I20250902 01:59:26.752116 1746 raft_consensus.cc:491] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:26.793457 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:26.730263 1842 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:26.799170 30926 rpcz_store.cc:276] 0902 01:59:24.608878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.609010 (+ 132us) service_pool.cc:225] Handling call
0902 01:59:24.613181 (+ 4171us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:24.680790 (+ 67609us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.729429 (+2048639us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b5400336654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:26.729705 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":151922560}
I20250902 01:59:26.792032 30967 leader_election.cc:304] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:26.800269 30965 leader_election.cc:304] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f
I20250902 01:59:26.798086 1761 raft_consensus.cc:2747] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:26.803320 31075 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:26.808300 31075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:26.772605 1883 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:26.806916 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:26.808772 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.225s user 0.017s sys 0.058s
I20250902 01:59:26.790529 1693 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:26.785951 1760 raft_consensus.cc:2747] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250902 01:59:26.796756 1840 raft_consensus.cc:3058] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:26.824985 30965 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:26.796167 1511 raft_consensus.cc:491] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:26.808980 31068 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.018s sys 0.061s
W20250902 01:59:26.831734 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.251s user 0.017s sys 0.058s
I20250902 01:59:26.835263 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:26.884379 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 1.31617e+06us
W20250902 01:59:26.832013 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.313s user 0.018s sys 0.060s
I20250902 01:59:26.815109 1849 raft_consensus.cc:2802] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election won for term 8
I20250902 01:59:26.861361 31724 raft_consensus.cc:1238] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 5. Current term is 6. Ops: []
I20250902 01:59:26.835774 1794 raft_consensus.cc:2802] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:26.885402 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.366s user 0.019s sys 0.060s
I20250902 01:59:26.681013 1877 consensus_queue.cc:260] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.864413 1701 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.865406 1751 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:26.791107 1770 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.865989 1693 raft_consensus.cc:695] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [term 8 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:26.805440 1799 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.823607 1692 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:26.837268 1831 consensus_meta.cc:287] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.051s user 0.008s sys 0.003s
W20250902 01:59:26.883775 31068 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.021s user 0.018s sys 0.061s
W20250902 01:59:26.883807 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 58928.7us
I20250902 01:59:26.906665 30965 leader_election.cc:304] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:26.789469 1804 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.801945 1651 raft_consensus.cc:513] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:26.887210 31722 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.258s user 0.014s sys 0.054s
I20250902 01:59:26.884616 31729 raft_consensus.cc:2466] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
W20250902 01:59:26.908609 31722 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.279s user 0.015s sys 0.054s
I20250902 01:59:26.889441 1794 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:26.830564 31040 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.001s sys 0.065s
I20250902 01:59:26.829340 1884 consensus_queue.cc:1046] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
W20250902 01:59:26.896811 31069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.115s user 0.006s sys 0.048s
I20250902 01:59:26.822566 1842 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:26.801697 1763 consensus_queue.cc:260] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.889214 1716 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250902 01:59:26.808516 1746 raft_consensus.cc:513] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:26.912708 31040 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.335s user 0.002s sys 0.065s
W20250902 01:59:26.914007 31069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.132s user 0.007s sys 0.048s
I20250902 01:59:26.894663 1849 raft_consensus.cc:695] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 8 LEADER]: Becoming Leader. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:26.901046 1701 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:26.907089 1368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.183s user 0.042s sys 0.155s
W20250902 01:59:26.906723 1808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.268s user 0.056s sys 0.149s
I20250902 01:59:26.927338 30892 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:26.908236 1761 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:26.892225 1511 raft_consensus.cc:513] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.911871 1877 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:26.909569 1328 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:26.914260 1794 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:26.939671 30923 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.062s user 0.011s sys 0.014s
W20250902 01:59:27.037557 30923 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.160s user 0.011s sys 0.014s
W20250902 01:59:26.999991 1808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.355s user 0.056s sys 0.149s
I20250902 01:59:26.912365 1874 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:26.967492 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:26.926004 1716 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:26.994891 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:26.909322 1692 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.930706 1815 raft_consensus.cc:2747] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250902 01:59:26.940011 1566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.054s user 0.004s sys 0.021s
I20250902 01:59:26.908152 1799 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:26.994032 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 66774us
I20250902 01:59:26.967182 1701 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.911190 1831 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:26.903100 1751 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:26.973551 1368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.250s user 0.042s sys 0.156s
W20250902 01:59:27.015913 1417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.013s sys 0.023s
W20250902 01:59:26.963562 30889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.125s user 0.000s sys 0.031s
I20250902 01:59:26.933776 1766 consensus_queue.cc:1046] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.097s
W20250902 01:59:27.043761 30994 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.000s sys 0.040s
W20250902 01:59:27.081671 30994 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s user 0.000s sys 0.040s
I20250902 01:59:27.049576 30998 raft_consensus.cc:2391] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 8: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:27.000699 1895 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.085474 31726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 553) took 5227 ms. Trace:
I20250902 01:59:27.032987 1761 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.040481 1787 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.097424 1751 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.100566 1692 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:27.080570 1417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.204s user 0.013s sys 0.024s
I20250902 01:59:27.103993 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 678) took 3203 ms. Trace:
I20250902 01:59:27.104440 31071 rpcz_store.cc:276] 0902 01:59:23.900311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:23.900456 (+ 145us) service_pool.cc:225] Handling call
0902 01:59:25.726944 (+1826488us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:25.727827 (+ 883us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.103752 (+1375925us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54000f32d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:27.103913 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4148919168}
I20250902 01:59:27.105796 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.106815 31071 raft_consensus.cc:2408] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for term 7 because replica has last-logged OpId of term: 6 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20250902 01:59:27.099252 31727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56978 (request call id 572) took 1786 ms. Trace:
I20250902 01:59:27.058133 1665 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:27.057875 1566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.180s user 0.004s sys 0.021s
I20250902 01:59:27.054137 1716 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:26.999511 1765 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:27.085951 31726 rpcz_store.cc:276] 0902 01:59:21.853355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:21.853484 (+ 129us) service_pool.cc:225] Handling call
0902 01:59:21.853810 (+ 326us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:25.276458 (+3422648us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.080228 (+1803770us) spinlock_profiling.cc:243] Waited 3.42 s on lock 0x7b540061f154. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:27.080431 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7864921472}
I20250902 01:59:26.934177 1893 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.108824 31727 rpcz_store.cc:276] 0902 01:59:25.312438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.497338 (+184900us) service_pool.cc:225] Handling call
0902 01:59:25.497716 (+ 378us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:25.512534 (+ 14818us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:25.512556 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:25.527305 (+ 14749us) log.cc:844] Serialized 26 byte log entry
0902 01:59:26.797342 (+1270037us) raft_consensus.cc:1643] Marking committed up to 5
0902 01:59:26.797395 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.797491 (+ 96us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:27.099032 (+301541us) raft_consensus.cc:1683] finished
0902 01:59:27.099055 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.099191 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41347,"raft.queue_time_us":18543,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":13533,"thread_start_us":1225475,"threads_started":1,"wal-append.queue_time_us":1447215}
I20250902 01:59:27.163094 31066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1961) took 1941 ms. Trace:
I20250902 01:59:27.087149 1766 consensus_queue.cc:1230] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250902 01:59:27.163602 31066 rpcz_store.cc:276] 0902 01:59:25.221339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.221483 (+ 144us) service_pool.cc:225] Handling call
0902 01:59:26.082220 (+860737us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:26.689230 (+607010us) raft_consensus.cc:1532] Early marking committed up to index 3
0902 01:59:26.689274 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:26.689332 (+ 58us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:26.689372 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.689454 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.162902 (+473448us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b5400201550. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:27.163028 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":356469,"raft.queue_time_us":400035,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":11943,"spinlock_wait_cycles":1978976000}
W20250902 01:59:27.114704 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.064595 1856 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:27.099207 1799 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.118233 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:27.118431 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:27.173349 31725 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:27.055243 1874 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.035339 1693 consensus_queue.cc:237] T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.076187 30889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.238s user 0.000s sys 0.031s
W20250902 01:59:27.115720 1781 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.046s user 0.010s sys 0.032s
W20250902 01:59:27.183169 1781 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.114s user 0.010s sys 0.032s
I20250902 01:59:27.183799 1781 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.184362 1781 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.186522 1781 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.188035 1368 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.188659 1368 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.165249 31066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.188187 1906 log.cc:927] Time spent T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20250902 01:59:27.193240 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 686) took 1537 ms. Trace:
I20250902 01:59:27.193559 31068 rpcz_store.cc:276] 0902 01:59:25.656081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.830685 (+174604us) service_pool.cc:225] Handling call
0902 01:59:25.830949 (+ 264us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:25.831115 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:25.831134 (+ 19us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:25.831513 (+ 379us) log.cc:844] Serialized 26 byte log entry
0902 01:59:26.906004 (+1074491us) raft_consensus.cc:1643] Marking committed up to 10
0902 01:59:26.910792 (+ 4788us) log.cc:844] Serialized 14 byte log entry
0902 01:59:26.913711 (+ 2919us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.913785 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:27.101069 (+187284us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b18001f2a60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1ce8bf 00007fc5ff1d0b61 00007fc5ff1d0b19 00007fc5ff1d0aa9 00007fc5ff1d0a71 00007fc5ff1cfd6d 0000000000379254 0000000000379099 00007fc5ff1b1b1a
0902 01:59:27.192973 (+ 91904us) raft_consensus.cc:1683] finished
0902 01:59:27.193073 (+ 100us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.193186 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2600,"prepare.queue_time_us":23598,"prepare.run_wall_time_us":29231,"raft.queue_time_us":10777,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":10417,"spinlock_wait_cycles":77710336,"thread_start_us":1043584,"threads_started":1,"wal-append.queue_time_us":1186085}
I20250902 01:59:27.195333 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.166525 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:27.196532 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.068794 1892 consensus_queue.cc:1046] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20250902 01:59:27.196724 30917 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:27.202433 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.231824 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.125818 1765 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.065162 1849 consensus_queue.cc:237] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.239135 1770 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.110126 1895 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.253396 31074 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.193s user 0.013s sys 0.062s
I20250902 01:59:27.190927 1368 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:27.258010 31074 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.217s user 0.013s sys 0.062s
I20250902 01:59:27.196589 1856 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.197500 1842 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.236402 1828 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:27.178177 1893 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.260144 1701 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.260473 1770 raft_consensus.cc:491] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.289230 1828 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.289738 1770 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.290663 1893 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.291113 1828 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.245945 1765 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:27.291644 1770 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:27.262848 1812 consensus_meta.cc:287] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 1.249s user 0.007s sys 0.002s
I20250902 01:59:27.291134 1856 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.292328 1812 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.293620 1856 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:27.294098 1812 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 10 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:27.294253 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:27.294806 31068 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 3 FOLLOWER]: Advancing to term 5
W20250902 01:59:27.297863 1899 log.cc:927] Time spent T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20250902 01:59:27.124994 1665 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.298043 1776 raft_consensus.cc:3053] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:27.298429 1776 raft_consensus.cc:738] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:27.267916 1886 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:27.291355 1687 consensus_meta.cc:287] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.506s user 0.005s sys 0.001s
I20250902 01:59:27.303717 31068 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:27.306052 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.263118 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:27.300376 31067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1950) took 2680 ms (client timeout 2159 ms). Trace:
W20250902 01:59:27.397241 31067 rpcz_store.cc:269] 0902 01:59:24.620423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.621019 (+ 596us) service_pool.cc:225] Handling call
0902 01:59:27.300214 (+2679195us) spinlock_profiling.cc:243] Waited 2.67 s on lock 0x7b54000f46d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:27.300320 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6151512960}
I20250902 01:59:27.399138 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.239085 1779 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.217s user 0.013s sys 0.038s
W20250902 01:59:27.307124 30995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.011s sys 0.024s
I20250902 01:59:27.240306 1566 raft_consensus.cc:491] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.258877 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1956) took 2333 ms. Trace:
W20250902 01:59:27.384734 31038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.014s sys 0.027s
I20250902 01:59:27.247494 1651 leader_election.cc:290] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.299444 1665 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:27.411648 30995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.012s sys 0.025s
W20250902 01:59:27.428855 31723 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.009s sys 0.020s
W20250902 01:59:27.429180 31723 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.009s sys 0.020s
W20250902 01:59:27.391782 31041 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.017s sys 0.027s
W20250902 01:59:27.430087 31041 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.017s sys 0.028s
W20250902 01:59:27.415619 31038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s user 0.014s sys 0.027s
I20250902 01:59:27.415354 31074 rpcz_store.cc:276] 0902 01:59:24.925511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.925754 (+ 243us) service_pool.cc:225] Handling call
0902 01:59:25.194371 (+268617us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:25.194569 (+ 198us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:26.040988 (+846419us) log.cc:844] Serialized 14 byte log entry
0902 01:59:27.258453 (+1217465us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:27.258516 (+ 63us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:27.258592 (+ 76us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.258684 (+ 92us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.258818 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1677886,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":2582,"raft.queue_time_us":78,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":2014081,"threads_started":2}
I20250902 01:59:27.420464 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:27.425863 31695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.007s sys 0.021s
I20250902 01:59:27.299129 1776 consensus_queue.cc:260] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.432191 31067 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:27.430722 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:27.448747 31695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.007s sys 0.021s
W20250902 01:59:27.411417 1779 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.392s user 0.013s sys 0.038s
I20250902 01:59:27.473171 31068 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2925593648194df5b7741d448a047401"
dest_uuid: "088424002a894fd5bcc11a06af111445"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:38110
W20250902 01:59:27.474550 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.471771 1406 raft_consensus.cc:3053] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:27.472302 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.393015 1770 raft_consensus.cc:491] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.413157 1746 leader_election.cc:290] T 744f11902df74dea8914fdd4a992259d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.412922 1511 leader_election.cc:290] T 818cd33c83fe427a981bf0c807af5b14 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:27.483814 30925 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.031s sys 0.054s
W20250902 01:59:27.516556 30925 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.031s sys 0.054s
W20250902 01:59:27.484112 31722 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.042s sys 0.043s
W20250902 01:59:27.486554 31728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.002s sys 0.033s
W20250902 01:59:27.518075 31728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.653s user 0.002s sys 0.033s
I20250902 01:59:27.302387 1886 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.513089 30923 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 655) took 3286 ms. Trace:
I20250902 01:59:27.447254 1822 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:27.528906 30923 rpcz_store.cc:276] 0902 01:59:24.226173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.226304 (+ 131us) service_pool.cc:225] Handling call
0902 01:59:24.291902 (+ 65598us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:24.300205 (+ 8303us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:24.300247 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:24.304352 (+ 4105us) log.cc:844] Serialized 26 byte log entry
0902 01:59:25.877022 (+1572670us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b180044bf00. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171 00007fc60210309f
0902 01:59:27.037975 (+1160953us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:27.038028 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.038111 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:27.512791 (+474680us) raft_consensus.cc:1683] finished
0902 01:59:27.512819 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.513023 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19446,"raft.queue_time_us":9673,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":29744,"spinlock_wait_cycles":5205376,"thread_start_us":1160749,"threads_started":1,"wal-append.queue_time_us":3089483}
W20250902 01:59:27.514394 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.386672 1687 raft_consensus.cc:513] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:27.519018 31722 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.608s user 0.042s sys 0.043s
I20250902 01:59:27.446979 1860 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:27.447122 1894 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.480253 1701 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.480933 1406 raft_consensus.cc:738] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
W20250902 01:59:27.548360 31038 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.516955 1514 consensus_queue.cc:1035] T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20250902 01:59:27.516815 1595 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.558439 1595 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.559329 31072 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:27.560357 1687 leader_election.cc:290] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:27.513713 1412 raft_consensus.cc:491] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.564141 1412 raft_consensus.cc:513] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.566344 1412 leader_election.cc:290] T 7f650d4e1b5f47febbda93daccba0e4f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.554368 1761 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.429873 1566 raft_consensus.cc:513] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.531925 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:27.566780 1765 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.485906 1776 raft_consensus.cc:3058] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:27.483361 1831 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.554184 1720 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:27.549352 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 680) took 1794 ms. Trace:
W20250902 01:59:27.549867 31692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.019s sys 0.023s
I20250902 01:59:27.553634 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 1022) took 3025 ms. Trace:
W20250902 01:59:27.603868 31692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.019s sys 0.024s
W20250902 01:59:27.605263 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 720245us
W20250902 01:59:27.607800 1901 log.cc:927] Time spent T 26ad5d89b038487b8bbc776913d5a215 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.353s user 0.001s sys 0.000s
W20250902 01:59:27.608995 1902 log.cc:927] Time spent T 818cd33c83fe427a981bf0c807af5b14 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.407s user 0.000s sys 0.000s
I20250902 01:59:27.609687 1804 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:27.611346 1903 log.cc:927] Time spent T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.378s user 0.000s sys 0.000s
I20250902 01:59:27.603804 1514 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:27.612377 30920 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57742 (request call id 675) took 1907 ms (client timeout 1766 ms). Trace:
I20250902 01:59:27.611996 1720 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.612151 1514 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.612753 30920 rpcz_store.cc:269] 0902 01:59:25.705040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.705180 (+ 140us) service_pool.cc:225] Handling call
0902 01:59:27.612176 (+1906996us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54000604d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:27.612305 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":943536384}
I20250902 01:59:27.614293 1514 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.615883 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46556 (request call id 1934) took 4061 ms. Trace:
I20250902 01:59:27.616209 31069 rpcz_store.cc:276] 0902 01:59:23.554186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:23.554287 (+ 101us) service_pool.cc:225] Handling call
0902 01:59:23.554549 (+ 262us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:23.656244 (+101695us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:23.656288 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:23.656801 (+ 513us) log.cc:844] Serialized 26 byte log entry
0902 01:59:25.781464 (+2124663us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b18004c4e60. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171 00007fc60210309f
0902 01:59:26.928230 (+1146766us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:26.928281 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:26.928373 (+ 92us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:27.615605 (+687232us) raft_consensus.cc:1683] finished
0902 01:59:27.615626 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.615831 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11429,"raft.queue_time_us":154841,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":15220,"spinlock_wait_cycles":5503232,"thread_start_us":1242347,"threads_started":2,"wal-append.queue_time_us":3398879}
I20250902 01:59:27.617570 1898 consensus_queue.cc:1035] T 39363bee7fbd45fdaa81366e5da31356 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250902 01:59:27.511808 1770 raft_consensus.cc:513] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.618002 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.619113 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.620255 31069 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:27.619596 30925 rpcz_store.cc:276] 0902 01:59:25.755227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.755346 (+ 119us) service_pool.cc:225] Handling call
0902 01:59:25.755603 (+ 257us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:25.755778 (+ 175us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:26.789078 (+1033300us) log.cc:844] Serialized 14 byte log entry
0902 01:59:27.548924 (+759846us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:27.548987 (+ 63us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:27.549042 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.549148 (+ 106us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.549277 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31044,"prepare.queue_time_us":1178829,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":31157,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":17645,"thread_start_us":1731570,"threads_started":2}
I20250902 01:59:27.621541 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.622341 31069 raft_consensus.cc:2466] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:27.623507 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.623468 31692 leader_election.cc:304] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:27.625135 31695 leader_election.cc:304] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:27.626879 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.549623 1828 raft_consensus.cc:491] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.549217 1856 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:27.629914 1834 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.000s sys 0.029s
W20250902 01:59:27.630321 1834 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.696s user 0.000s sys 0.029s
I20250902 01:59:27.630528 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.633064 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:27.571848 1412 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:27.619310 1770 leader_election.cc:290] T 39363bee7fbd45fdaa81366e5da31356 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.560369 1595 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:27.644381 30997 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.031s sys 0.083s
W20250902 01:59:27.646154 30997 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.031s sys 0.083s
I20250902 01:59:27.631472 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.640851 30967 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.006s sys 0.024s
I20250902 01:59:27.642661 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.649312 30967 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s user 0.006s sys 0.024s
I20250902 01:59:27.557073 1822 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:27.650125 30996 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.031s sys 0.083s
W20250902 01:59:27.650518 30996 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.031s sys 0.083s
I20250902 01:59:27.632242 1664 raft_consensus.cc:2747] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:27.645038 31692 leader_election.cc:304] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:27.646456 30965 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.011s sys 0.019s
I20250902 01:59:27.639302 1874 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:27.574029 1894 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.574579 1701 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.654227 30965 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s user 0.011s sys 0.020s
W20250902 01:59:27.669486 31693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.017s sys 0.017s
W20250902 01:59:27.671110 31693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.801s user 0.017s sys 0.017s
W20250902 01:59:27.671329 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 801538us
I20250902 01:59:27.643793 1514 raft_consensus.cc:2802] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:27.636709 1856 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.629022 1898 consensus_queue.cc:1046] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
W20250902 01:59:27.655092 31001 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.004s sys 0.057s
W20250902 01:59:27.678260 31001 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.004s sys 0.057s
I20250902 01:59:27.675057 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:27.681893 30965 connection.cc:232] RPC call timeout handler was delayed by -2.0278794890000427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20250902 01:59:27.652153 1760 raft_consensus.cc:2747] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250902 01:59:27.661556 1788 consensus_meta.cc:287] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 1.545s user 0.008s sys 0.003s
I20250902 01:59:27.639612 31067 raft_consensus.cc:2466] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:27.692376 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:27.637018 1828 raft_consensus.cc:513] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.696146 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:27.696897 30919 raft_consensus.cc:2391] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 5: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:27.697700 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:27.698027 1828 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.698158 30919 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:27.700726 30918 raft_consensus.cc:1238] T 703afdc575144ab78740c24ef2852ca8 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: []
W20250902 01:59:27.695217 30966 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.006s sys 0.016s
I20250902 01:59:27.640574 1412 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.708562 31039 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.017s sys 0.026s
I20250902 01:59:27.709057 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:27.709057 31039 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.017s sys 0.026s
W20250902 01:59:27.681105 1716 consensus_meta.cc:287] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.000s
I20250902 01:59:27.699383 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:27.717816 30925 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:27.700287 30924 raft_consensus.cc:1238] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:27.714998 31000 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1989) took 1774 ms. Trace:
W20250902 01:59:27.727669 1792 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.003s sys 0.024s
W20250902 01:59:27.732738 1792 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s user 0.003s sys 0.024s
I20250902 01:59:27.641999 1406 consensus_queue.cc:260] T aa4b903a93464946ae238ce634b483d0 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:27.736626 1911 log.cc:927] Time spent T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20250902 01:59:27.737937 1913 log.cc:927] Time spent T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250902 01:59:27.739603 1909 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.740233 1909 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.699924 1788 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.672806 1770 raft_consensus.cc:491] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.676009 1665 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:27.707993 30967 leader_election.cc:304] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:27.707379 30966 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.007s sys 0.016s
I20250902 01:59:27.555222 1860 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:27.700691 31067 raft_consensus.cc:2408] T 558db87de6fa4361bf0ca91e27d34943 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250902 01:59:27.672256 1664 raft_consensus.cc:2747] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:27.712334 1842 raft_consensus.cc:491] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:27.713361 1417 raft_consensus.cc:3053] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:27.713802 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:27.714704 30926 raft_consensus.cc:1238] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250902 01:59:27.673075 1514 raft_consensus.cc:491] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
W20250902 01:59:27.676806 30993 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.008s sys 0.012s
I20250902 01:59:27.678990 1874 raft_consensus.cc:491] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:27.700275 31069 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:27.728188 30923 raft_consensus.cc:1273] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Refusing update from remote peer 088424002a894fd5bcc11a06af111445: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250902 01:59:27.782724 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.863294 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:28.179515 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 471578us
I20250902 01:59:28.179862 30994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 1990) took 2238 ms. Trace:
W20250902 01:59:28.181425 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:27.824404 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:27.787882 30921 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.017s sys 0.029s
W20250902 01:59:28.182309 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:28.182480 30921 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.090s user 0.028s sys 0.038s
W20250902 01:59:27.764657 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 66438us
I20250902 01:59:28.183964 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:28.185170 31067 raft_consensus.cc:2466] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
W20250902 01:59:28.185177 30993 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s user 0.009s sys 0.014s
I20250902 01:59:27.679234 1789 raft_consensus.cc:491] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.188093 31001 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53772 (request call id 2047) took 1275 ms. Trace:
I20250902 01:59:27.797573 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:28.196770 30920 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:28.179417 31074 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:27.731611 31000 rpcz_store.cc:276] 0902 01:59:25.941075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.945280 (+ 4205us) service_pool.cc:225] Handling call
0902 01:59:25.945583 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:25.945818 (+ 235us) raft_consensus.cc:1532] Early marking committed up to index 7
0902 01:59:25.981628 (+ 35810us) log.cc:844] Serialized 14 byte log entry
0902 01:59:27.229860 (+1248232us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b1800344140. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d742da 00007fc60213a9da
0902 01:59:27.714727 (+484867us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:27.714770 (+ 43us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:27.714805 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.714854 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.714946 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5647,"prepare.queue_time_us":35554,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":5728,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":23234304,"thread_start_us":478948,"threads_started":1}
I20250902 01:59:28.186805 30993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 544) took 2674 ms. Trace:
I20250902 01:59:28.180383 30994 rpcz_store.cc:276] 0902 01:59:25.941364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.941677 (+ 313us) service_pool.cc:225] Handling call
0902 01:59:25.942009 (+ 332us) raft_consensus.cc:1469] Updating replica for 3 ops
0902 01:59:25.942391 (+ 382us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:25.942417 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0902 01:59:25.960038 (+ 17621us) log.cc:844] Serialized 78 byte log entry
0902 01:59:27.082112 (+1122074us) raft_consensus.cc:1643] Marking committed up to 2
0902 01:59:27.082164 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.082250 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:28.179606 (+1097356us) raft_consensus.cc:1683] finished
0902 01:59:28.179633 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.179799 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4271,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":4499,"thread_start_us":1090912,"threads_started":1,"wal-append.queue_time_us":1585795}
I20250902 01:59:28.186422 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
W20250902 01:59:28.199741 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.200780 30889 leader_election.cc:304] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
W20250902 01:59:28.201309 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:27.693483 1804 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.201469 30993 rpcz_store.cc:276] 0902 01:59:25.512529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:25.512688 (+ 159us) service_pool.cc:225] Handling call
0902 01:59:25.513067 (+ 379us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:25.518425 (+ 5358us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:25.518448 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:25.527189 (+ 8741us) log.cc:844] Serialized 26 byte log entry
0902 01:59:27.107118 (+1579929us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b180036d3c0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc607e73e31 00007fc607e80781 00007fc5ff1b10f7 00007fc607decb89 00007fc5ff1cda4b 00007fc5ff1cbcef 00007fc5ff1cdfb2 00007fc601d6e2bb 00007fc601d73dc3 00007fc601d74171 00007fc60210309f
0902 01:59:28.186121 (+1079003us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:28.186173 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:28.186387 (+ 214us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:28.186510 (+ 123us) raft_consensus.cc:1683] finished
0902 01:59:28.186521 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.186701 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4970,"raft.queue_time_us":5510,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":19837,"spinlock_wait_cycles":24763648,"thread_start_us":1078834,"threads_started":1,"wal-append.queue_time_us":2323660}
W20250902 01:59:27.971071 1693 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.017s sys 0.013s
I20250902 01:59:27.707420 1746 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
W20250902 01:59:27.990700 1916 log.cc:927] Time spent T 69f27fd54f16465fa39cc86d8d6c07bc P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.281s user 0.000s sys 0.000s
I20250902 01:59:28.208035 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:28.155956 1877 consensus_meta.cc:287] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.117s user 0.009s sys 0.001s
W20250902 01:59:28.154904 1759 consensus_meta.cc:287] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 2.062s user 0.011s sys 0.000s
W20250902 01:59:28.032132 1883 consensus_meta.cc:287] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.168s user 0.007s sys 0.003s
I20250902 01:59:28.211817 30998 raft_consensus.cc:1238] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:28.212234 31693 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:28.212535 30997 raft_consensus.cc:1238] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250902 01:59:28.211815 31728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 1036) took 1379 ms. Trace:
I20250902 01:59:27.706530 1803 raft_consensus.cc:491] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.206600 1761 raft_consensus.cc:491] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.313040 31726 raft_consensus.cc:1273] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20250902 01:59:28.313395 31726 pending_rounds.cc:85] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Aborting all ops after (but not including) 6
I20250902 01:59:27.799268 1716 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:27.709971 1892 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:28.188154 1802 consensus_meta.cc:287] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 2.028s user 0.012s sys 0.000s
I20250902 01:59:28.221385 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:27.716171 1919 raft_consensus.cc:491] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:27.710155 1895 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.173106 1925 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.365413 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.366859 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 547) took 1578 ms. Trace:
I20250902 01:59:28.352898 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.224272 31067 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:28.225271 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:28.160007 1701 leader_election.cc:290] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:28.161217 1822 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:28.009255 1566 leader_election.cc:290] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:27.743526 1788 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 10 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.033632 1412 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.184232 1417 raft_consensus.cc:738] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:28.183636 1905 raft_consensus.cc:491] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.159256 1894 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.187043 1514 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:28.194865 1893 raft_consensus.cc:2802] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:28.292273 30922 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:38794 (request call id 1166) took 2191 ms. Trace:
I20250902 01:59:28.186296 1781 raft_consensus.cc:491] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:28.213310 1693 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s user 0.021s sys 0.017s
I20250902 01:59:28.299150 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.384508 30922 rpcz_store.cc:276] 0902 01:59:26.036069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.039102 (+ 3033us) service_pool.cc:225] Handling call
0902 01:59:28.226697 (+2187595us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54002a12d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b4cd0 00007fc602138eea 00007fc60709160e 00007fc60072ecc7 00007fc60072ec52 00007fc60072eb81 00007fc60072eafc 00007fc60072ddb2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:28.227234 (+ 537us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3471327616}
I20250902 01:59:28.216001 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.303150 31728 rpcz_store.cc:276] 0902 01:59:26.832087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.833067 (+ 980us) service_pool.cc:225] Handling call
0902 01:59:26.864539 (+ 31472us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:26.864758 (+ 219us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:27.518529 (+653771us) log.cc:844] Serialized 14 byte log entry
0902 01:59:28.211496 (+692967us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:28.211568 (+ 72us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:28.211619 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:28.211670 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.211765 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":240088,"prepare.queue_time_us":831011,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":240558,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":12157,"thread_start_us":1105870,"threads_started":2}
W20250902 01:59:28.313071 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 100900us
I20250902 01:59:28.337172 31726 pending_rounds.cc:107] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20250902 01:59:28.332594 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.335041 31729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.386659 31070 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:27.742517 1909 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.345050 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.226328 1664 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:28.347335 31724 raft_consensus.cc:1273] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20250902 01:59:28.389217 31693 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
W20250902 01:59:28.368677 1840 consensus_meta.cc:287] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 1.479s user 0.011s sys 0.000s
W20250902 01:59:28.349751 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 164638us
I20250902 01:59:28.282459 1746 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:28.299516 1776 consensus_meta.cc:287] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.687s user 0.009s sys 0.000s
W20250902 01:59:28.305526 1923 log.cc:927] Time spent T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250902 01:59:28.205487 1874 raft_consensus.cc:513] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:28.195693 1770 raft_consensus.cc:513] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.436128 1886 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.222976 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.436743 1770 leader_election.cc:290] T 331364bf121c480483b11ff32993e1ec P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.207356 1789 raft_consensus.cc:513] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.436964 31721 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:28.438594 1789 leader_election.cc:290] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.313587 1877 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.439241 1877 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:28.440081 1936 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250902 01:59:28.441099 1877 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.209254 1842 raft_consensus.cc:513] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:28.346206 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 481663us
I20250902 01:59:28.387241 1834 raft_consensus.cc:491] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.384318 1566 raft_consensus.cc:2747] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250902 01:59:28.367058 31730 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:28.367367 30995 rpcz_store.cc:276] 0902 01:59:26.788017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.788170 (+ 153us) service_pool.cc:225] Handling call
0902 01:59:26.788532 (+ 362us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:26.794440 (+ 5908us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:26.794479 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:26.794968 (+ 489us) log.cc:844] Serialized 26 byte log entry
0902 01:59:27.429235 (+634267us) raft_consensus.cc:1643] Marking committed up to 4
0902 01:59:27.429286 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.429368 (+ 82us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:28.366657 (+937289us) raft_consensus.cc:1683] finished
0902 01:59:28.366680 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.366807 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5931,"raft.queue_time_us":9272,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":633953,"threads_started":1,"wal-append.queue_time_us":874077}
I20250902 01:59:28.293251 1915 raft_consensus.cc:491] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:28.305240 1831 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.382375 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:28.383422 31000 raft_consensus.cc:2466] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:28.306039 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.385428 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:28.386076 31726 raft_consensus.cc:2887] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:28.460825 30999 raft_consensus.cc:3058] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:28.388037 30997 raft_consensus.cc:2408] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:28.220647 1804 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.342988 1830 raft_consensus.cc:3053] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:28.202042 1665 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:28.480409 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 91015.2us
I20250902 01:59:28.423846 1411 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250902 01:59:28.388288 1794 consensus_meta.cc:287] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 1.336s user 0.005s sys 0.006s
I20250902 01:59:28.381124 1778 raft_consensus.cc:491] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:28.347096 1761 raft_consensus.cc:513] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.341342 1784 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:28.443990 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 687) took 2783 ms. Trace:
I20250902 01:59:28.389775 1905 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.458019 31038 leader_election.cc:304] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters: 975ecd397f5146409baac87e56a302e9
I20250902 01:59:28.365897 1892 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.473917 1834 raft_consensus.cc:513] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:28.605218 1892 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.317915 1883 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.606195 31071 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 9: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:28.606297 30926 raft_consensus.cc:1238] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:28.390662 1788 raft_consensus.cc:491] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:28.473387 1417 consensus_queue.cc:260] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.419286 1701 raft_consensus.cc:3058] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:28.340122 1803 raft_consensus.cc:513] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.478971 30999 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:28.442857 1842 leader_election.cc:290] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.483419 30967 leader_election.cc:304] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:28.613940 30967 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 130595us
W20250902 01:59:28.614153 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 413382us
I20250902 01:59:28.611967 30992 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 7.
I20250902 01:59:28.603000 1514 raft_consensus.cc:513] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.457366 1693 raft_consensus.cc:491] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:28.469146 1893 raft_consensus.cc:491] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:28.470463 1884 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.474673 1915 raft_consensus.cc:513] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:28.533007 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.365897 1802 raft_consensus.cc:513] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } attrs { replace: true } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.382314 1925 raft_consensus.cc:513] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.419909 1781 raft_consensus.cc:513] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.422935 1664 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.542749 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:28.539826 30924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:28.519182 1789 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:28.571691 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.590150 30917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:28.545300 31074 rpcz_store.cc:276] 0902 01:59:25.661059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.389474 (+2728415us) service_pool.cc:225] Handling call
0902 01:59:28.443451 (+ 53977us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:28.443696 (+ 245us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:28.443720 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:28.443761 (+ 41us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:28.443790 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:28.443840 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.443945 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250902 01:59:28.591307 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.528508 1566 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
W20250902 01:59:28.583287 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.471184 1886 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:28.593225 30919 raft_consensus.cc:1238] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 4. Current term is 5. Ops: []
W20250902 01:59:28.597193 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 139230us
I20250902 01:59:28.589888 1784 raft_consensus.cc:491] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:28.592667 1778 raft_consensus.cc:513] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } attrs { replace: true } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.485097 1877 raft_consensus.cc:491] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:28.477195 1830 raft_consensus.cc:738] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:28.607122 31071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1774) took 2766 ms (client timeout 3150 ms). Trace:
I20250902 01:59:28.607770 31067 raft_consensus.cc:1238] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 9. Ops: []
I20250902 01:59:28.607228 1894 raft_consensus.cc:2802] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:28.364163 1919 raft_consensus.cc:513] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.532001 1794 raft_consensus.cc:513] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.606896 1788 raft_consensus.cc:513] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.583122 1411 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.481696 1831 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.607138 1417 raft_consensus.cc:3058] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:28.826251 30924 raft_consensus.cc:3058] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:28.655901 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:28.787796 30965 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 77522.7us
I20250902 01:59:28.670744 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.195992 31001 rpcz_store.cc:276] 0902 01:59:26.913060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.913193 (+ 133us) service_pool.cc:225] Handling call
0902 01:59:26.913549 (+ 356us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:26.913783 (+ 234us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:26.913818 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:26.923514 (+ 9696us) log.cc:844] Serialized 26 byte log entry
0902 01:59:27.678744 (+755230us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:28.105576 (+426832us) log.cc:844] Serialized 14 byte log entry
0902 01:59:28.187781 (+ 82205us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:28.187865 (+ 84us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:28.187908 (+ 43us) raft_consensus.cc:1683] finished
0902 01:59:28.187923 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.188041 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78120,"raft.queue_time_us":85700,"thread_start_us":1181438,"threads_started":2,"wal-append.queue_time_us":1094620}
W20250902 01:59:28.665459 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 1.29926e+06us
I20250902 01:59:28.621656 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.658445 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.658150 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.653573 30921 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57742 (request call id 682) took 2471 ms. Trace:
I20250902 01:59:28.652180 31730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:28.658825 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.654152 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:28.615103 30892 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:28.715890 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.615690 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.653875 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d8ce2ebbb1c24a6c998649ed3656707a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:28.594401 1860 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:28.837339 30889 connection.cc:232] RPC call timeout handler was delayed by -1.4224482470000339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20250902 01:59:28.609719 1946 log.cc:927] Time spent T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20250902 01:59:28.838346 31071 rpcz_store.cc:269] 0902 01:59:25.840190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:27.107445 (+1267255us) service_pool.cc:225] Handling call
0902 01:59:28.606960 (+1499515us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400257954. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:28.607074 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3152124160}
W20250902 01:59:28.654439 1940 log.cc:927] Time spent T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
W20250902 01:59:28.654078 1938 log.cc:927] Time spent T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20250902 01:59:28.865516 31075 raft_consensus.cc:2366] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 7. Current term is 8.
W20250902 01:59:28.867834 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:28.871394 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:28.873235 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.874714 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.655419 1883 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.622161 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:28.877344 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:28.876224 1716 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.033109 30995 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:29.034900 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.035828 30995 raft_consensus.cc:2466] T 24426e817ea94ac5a7014c3457ea087c P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:28.651602 1761 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:28.657660 1905 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.037110 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.023953 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45756 (request call id 697) took 1452 ms. Trace:
I20250902 01:59:28.836970 1784 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.038221 31069 rpcz_store.cc:276] 0902 01:59:27.571082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:27.631299 (+ 60217us) service_pool.cc:225] Handling call
0902 01:59:29.023781 (+1392482us) spinlock_profiling.cc:243] Waited 49.9 ms on lock 0x7b54000406d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.023886 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114758528}
W20250902 01:59:29.033226 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.046622 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba8e025cdf94c449fa56870398f3419" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.047428 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.048086 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.048449 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.049064 30919 raft_consensus.cc:2391] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 7: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
I20250902 01:59:29.037133 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.016108 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.052702 30996 raft_consensus.cc:2391] T 9685199b6c7b42f391ab48fdf86d71ad P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 5: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:29.042721 30993 raft_consensus.cc:2408] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20250902 01:59:29.039141 30965 leader_election.cc:336] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457): Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.861s (SENT)
I20250902 01:59:28.830803 1893 raft_consensus.cc:3058] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.040510 30997 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:28.657155 1665 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:29.036145 31039 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:44457 timed out after 1.920s (SENT) [suppressed 20 similar messages]
I20250902 01:59:29.057889 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.058113 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.058660 30923 raft_consensus.cc:2408] T 69f27fd54f16465fa39cc86d8d6c07bc P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250902 01:59:29.054262 30999 raft_consensus.cc:2408] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20250902 01:59:29.037214 30921 rpcz_store.cc:276] 0902 01:59:26.182227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.183005 (+ 778us) service_pool.cc:225] Handling call
0902 01:59:26.183408 (+ 403us) raft_consensus.cc:1469] Updating replica for 1 ops
0902 01:59:26.183676 (+ 268us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:26.183711 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0902 01:59:27.091932 (+908221us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b2c0004d918. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc600ad39a1 00007fc600ae2fd2 00007fc60216c021 00007fc6021303fa 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32
0902 01:59:27.092378 (+ 446us) log.cc:844] Serialized 26 byte log entry
0902 01:59:28.183469 (+1091091us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:28.183507 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:28.183822 (+ 315us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:28.653270 (+469448us) raft_consensus.cc:1683] finished
0902 01:59:28.653292 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:28.653504 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47474560,"thread_start_us":1090667,"threads_started":1,"wal-append.queue_time_us":1456487}
I20250902 01:59:29.049679 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.054378 30965 leader_election.cc:304] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:28.842077 1894 raft_consensus.cc:491] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250902 01:59:29.064266 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.064162 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:29.064453 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 449344us
I20250902 01:59:29.065201 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.061475 30919 raft_consensus.cc:2466] T 2e3ca4f287b94f60b51ef1768df7bb70 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 5.
I20250902 01:59:29.062104 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:29.062563 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.063212 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
W20250902 01:59:29.026185 1900 log.cc:927] Time spent T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 2.092s user 0.006s sys 0.017s
I20250902 01:59:29.066408 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.072543 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.073344 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.073640 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.067178 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.067190 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "558db87de6fa4361bf0ca91e27d34943" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.649303 1892 raft_consensus.cc:2802] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:29.064719 30923 raft_consensus.cc:2466] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 3.
I20250902 01:59:29.077540 30889 leader_election.cc:304] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.004479 1925 leader_election.cc:290] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.071475 31071 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:28.886085 1746 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.084720 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1770) took 3245 ms. Trace:
I20250902 01:59:29.085249 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "744f11902df74dea8914fdd4a992259d" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.085160 31071 rpcz_store.cc:276] 0902 01:59:25.839235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.038244 (+3199009us) service_pool.cc:225] Handling call
0902 01:59:29.071323 (+ 33079us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.084454 (+ 13131us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.084663 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:29.067647 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.075380 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9685199b6c7b42f391ab48fdf86d71ad" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.075891 30919 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.077015 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.087278 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24426e817ea94ac5a7014c3457ea087c" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:29.077620 30964 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.078832 30892 leader_election.cc:304] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:29.079547 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.080168 31723 raft_consensus.cc:1238] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:29.045197 1919 leader_election.cc:290] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.026916 1874 leader_election.cc:290] T 4120ecbc46e34f93bb86004798318c08 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.123026 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:28.885073 1792 consensus_queue.cc:1046] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.190s
I20250902 01:59:29.029183 1802 leader_election.cc:290] T 4ba5aa5dce214071a172ea365440f0dc P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:29.005489 1937 log.cc:927] Time spent T 8f5d81000607410da8880ef8e5517680 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.398s user 0.000s sys 0.000s
I20250902 01:59:29.081025 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.082194 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.082612 31721 raft_consensus.cc:2391] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 10: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:29.133311 30995 raft_consensus.cc:2466] T d7f66bcc25964883baf67f48889ec766 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:29.024940 1514 leader_election.cc:290] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:28.836741 1693 raft_consensus.cc:513] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.083467 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:28.843544 1789 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.017458 1794 leader_election.cc:290] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.018883 1915 leader_election.cc:290] T dad02ef06fef44c5b5ecc42134c43091 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:29.119585 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.151876 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.152707 31726 raft_consensus.cc:2466] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:29.143714 1860 consensus_queue.cc:1046] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20250902 01:59:29.158777 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.159514 30997 raft_consensus.cc:2466] T 349593d2d3574cd6ab281c105756f435 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 8.
I20250902 01:59:29.123734 30996 raft_consensus.cc:2391] T 2e3ca4f287b94f60b51ef1768df7bb70 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 6: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:29.164486 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.165603 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "331364bf121c480483b11ff32993e1ec" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.166210 30996 raft_consensus.cc:2466] T 331364bf121c480483b11ff32993e1ec P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 4.
I20250902 01:59:29.167150 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.123131 1760 raft_consensus.cc:491] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.175946 1960 raft_consensus.cc:491] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:29.176142 1760 raft_consensus.cc:513] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.176640 1960 raft_consensus.cc:513] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.178943 1760 leader_election.cc:290] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.081315 1892 raft_consensus.cc:491] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:29.179749 1964 raft_consensus.cc:491] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.179914 1892 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.180343 1964 raft_consensus.cc:513] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.182657 1964 leader_election.cc:290] T 63fa3c697d3e40afb9a01578cbe72700 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.184507 1962 raft_consensus.cc:3053] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:29.184588 1742 consensus_queue.cc:1035] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250902 01:59:29.185070 1962 raft_consensus.cc:738] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:29.127774 1955 raft_consensus.cc:2802] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:29.186106 1955 raft_consensus.cc:695] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:29.186252 1962 consensus_queue.cc:260] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.187670 1962 raft_consensus.cc:3058] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:29.187269 1955 consensus_queue.cc:237] T d8ce2ebbb1c24a6c998649ed3656707a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.179342 1960 leader_election.cc:290] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.129590 1934 raft_consensus.cc:491] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:29.081948 1761 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:29.195533 1934 raft_consensus.cc:513] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.195879 1761 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.198011 1934 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.198609 1761 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.199024 1664 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.201352 1664 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.201843 1664 raft_consensus.cc:513] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.204921 1778 leader_election.cc:290] T 92f16b8a15144fcf8807608bdee71123 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.205192 1664 leader_election.cc:290] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.091859 30892 leader_election.cc:304] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:29.071882 1894 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:28.872126 1877 raft_consensus.cc:513] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.131419 1741 raft_consensus.cc:491] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:29.208077 1741 raft_consensus.cc:513] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.210171 1741 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.127230 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.210939 30921 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:29.211939 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "349593d2d3574cd6ab281c105756f435" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.213075 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.213758 1877 leader_election.cc:290] T 703afdc575144ab78740c24ef2852ca8 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:29.215040 1803 leader_election.cc:290] T d0b2dd8f5f814a7d8b48a74020055750 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.218423 1894 raft_consensus.cc:513] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:29.220803 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 129017us
I20250902 01:59:29.088958 30964 leader_election.cc:304] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:28.872305 1886 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.132606 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d30dc90c304472ab905d3af8400b8d0" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.136040 31071 raft_consensus.cc:1238] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 5. Ops: []
I20250902 01:59:29.275424 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1779) took 3406 ms. Trace:
I20250902 01:59:29.275835 31071 rpcz_store.cc:276] 0902 01:59:25.869386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.086086 (+3216700us) service_pool.cc:225] Handling call
0902 01:59:29.135898 (+ 49812us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.275146 (+139248us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.275378 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:29.132073 1665 raft_consensus.cc:2747] T 22b1ce0ddf934a6baa56f3ee89e05309 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250902 01:59:29.136924 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.137588 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.135957 30923 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 4.
I20250902 01:59:28.837491 1884 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:28.950963 1566 raft_consensus.cc:513] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.144364 1720 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:29.144629 1952 log.cc:927] Time spent T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
I20250902 01:59:29.144996 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "818cd33c83fe427a981bf0c807af5b14" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.145859 1693 leader_election.cc:290] T 0692c5029b5e4ab088324d038ff964be P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.147010 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:29.315199 31001 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Advancing to term 10
I20250902 01:59:29.148921 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39363bee7fbd45fdaa81366e5da31356" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.318966 30993 raft_consensus.cc:2466] T 39363bee7fbd45fdaa81366e5da31356 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 9.
I20250902 01:59:29.151970 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7f66bcc25964883baf67f48889ec766" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.085824 31728 raft_consensus.cc:2466] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
W20250902 01:59:29.163036 1701 consensus_meta.cc:287] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.550s user 0.006s sys 0.000s
I20250902 01:59:29.154254 1874 raft_consensus.cc:491] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.020226 1788 leader_election.cc:290] T 6d30dc90c304472ab905d3af8400b8d0 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.184803 1792 raft_consensus.cc:3053] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:29.185496 1860 raft_consensus.cc:3053] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Stepping down as leader of term 7
I20250902 01:59:29.207247 1789 leader_election.cc:290] T f0f24bac96dc42dba9adc1f6679c0d48 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.213950 30921 raft_consensus.cc:2408] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250902 01:59:29.064769 1766 raft_consensus.cc:491] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:29.125459 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.348028 30922 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.268944 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:29.349731 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 80866.8us
I20250902 01:59:29.153646 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.185334 1742 consensus_queue.cc:1230] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer 4372b4c4c866431faa1bf3e91fd47a88 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250902 01:59:29.272627 31069 raft_consensus.cc:1238] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:29.219219 1781 leader_election.cc:290] T eecf760e0123488fb797b4457e0888d7 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.075223 1905 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:29.125422 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 48516.6us
I20250902 01:59:29.358287 30889 leader_election.cc:304] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.280481 30924 raft_consensus.cc:2466] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 10.
I20250902 01:59:29.084746 1716 raft_consensus.cc:2747] T cba8e025cdf94c449fa56870398f3419 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:29.360584 30926 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 4372b4c4c866431faa1bf3e91fd47a88 [term 10 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 9. Current term is 10. Ops: []
I20250902 01:59:29.088657 31725 raft_consensus.cc:2466] T 8f5d81000607410da8880ef8e5517680 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:29.284052 30923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:29.285238 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 401824us
W20250902 01:59:29.292074 30964 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:44285, user_credentials={real_user=slave}} blocked reactor thread for 203158us
I20250902 01:59:29.362651 30964 leader_election.cc:304] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 975ecd397f5146409baac87e56a302e9; no voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445
I20250902 01:59:29.362838 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38649bfc334c4db6921eb3b12130e10d" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:29.363024 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f27fd54f16465fa39cc86d8d6c07bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.312189 30920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.363881 31723 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.313762 31074 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:29.324183 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.330654 31693 leader_election.cc:304] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f; no voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88
I20250902 01:59:29.331400 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.365173 31038 leader_election.cc:304] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:29.365429 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.365621 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f650d4e1b5f47febbda93daccba0e4f" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.332063 1860 raft_consensus.cc:738] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:29.333317 1731 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.372177 30964 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters: 088424002a894fd5bcc11a06af111445
I20250902 01:59:29.312110 1741 raft_consensus.cc:2802] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:29.339308 1834 leader_election.cc:290] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:29.339529 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.374859 30889 leader_election.cc:304] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:29.341486 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 688) took 3677 ms. Trace:
I20250902 01:59:29.379818 31067 rpcz_store.cc:276] 0902 01:59:25.664195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.475139 (+2810944us) service_pool.cc:225] Handling call
0902 01:59:28.607615 (+132476us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:28.838759 (+231144us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.341228 (+502469us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400257950. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.341409 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":303723264}
I20250902 01:59:29.329470 1874 raft_consensus.cc:513] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.343083 30925 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 2013) took 1645 ms. Trace:
I20250902 01:59:29.329777 1792 raft_consensus.cc:738] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:27.603969 31722 rpcz_store.cc:276] 0902 01:59:24.528684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:24.528801 (+ 117us) service_pool.cc:225] Handling call
0902 01:59:24.548437 (+ 19636us) raft_consensus.cc:1469] Updating replica for 2 ops
0902 01:59:24.572221 (+ 23784us) raft_consensus.cc:1532] Early marking committed up to index 6
0902 01:59:25.577519 (+1005298us) log.cc:844] Serialized 14 byte log entry
0902 01:59:26.909132 (+1331613us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0902 01:59:26.910440 (+ 1308us) log.cc:844] Serialized 52 byte log entry
0902 01:59:26.910846 (+ 406us) raft_consensus.cc:1643] Marking committed up to 7
0902 01:59:27.552937 (+642091us) log.cc:844] Serialized 14 byte log entry
0902 01:59:27.553238 (+ 301us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:27.553329 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0902 01:59:27.553378 (+ 49us) raft_consensus.cc:1683] finished
0902 01:59:27.553395 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:27.553574 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28008,"prepare.queue_time_us":1377416,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":157347,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":71105,"thread_start_us":2880190,"threads_started":3,"wal-append.queue_time_us":1446236}
I20250902 01:59:29.385444 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:29.385418 1792 consensus_queue.cc:260] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.386107 31722 raft_consensus.cc:3058] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.386600 1792 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:29.346696 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.374269 1781 consensus_queue.cc:1035] T 0692c5029b5e4ab088324d038ff964be P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20250902 01:59:29.388396 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.376102 1741 raft_consensus.cc:491] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.389107 1799 raft_consensus.cc:491] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.389230 1963 raft_consensus.cc:2747] T 744f11902df74dea8914fdd4a992259d P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250902 01:59:29.389670 1799 raft_consensus.cc:513] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.390489 1884 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.390636 31068 raft_consensus.cc:604] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.391039 31068 raft_consensus.cc:612] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:29.327493 1701 raft_consensus.cc:2747] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:29.391556 31066 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:29.379163 30889 leader_election.cc:304] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.389146 31728 raft_consensus.cc:2408] T aeb7c457a3f741b5b4d057f06363c614 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20250902 01:59:29.395380 31722 raft_consensus.cc:2466] T 2925593648194df5b7741d448a047401 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:29.397229 31722 raft_consensus.cc:1238] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 10 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 9. Current term is 10. Ops: []
I20250902 01:59:29.354640 31001 raft_consensus.cc:2466] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 10.
I20250902 01:59:29.356123 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.269738 1954 raft_consensus.cc:2802] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:29.366024 31725 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:29.366957 31038 leader_election.cc:304] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
W20250902 01:59:29.367820 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 302287us
I20250902 01:59:29.369189 31721 raft_consensus.cc:2391] T 5a3db1aaea8d4a3eba2de840361e0849 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:29.366357 31723 raft_consensus.cc:2466] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:29.369375 31693 leader_election.cc:304] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:29.387938 1892 raft_consensus.cc:513] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.376398 31726 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.381297 30925 rpcz_store.cc:276] 0902 01:59:27.697854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:27.719291 (+ 21437us) service_pool.cc:225] Handling call
0902 01:59:27.719510 (+ 219us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:28.537501 (+817991us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:28.537528 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:28.537578 (+ 50us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:28.537612 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.342861 (+805249us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b54002409d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:29.342907 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:29.343029 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1880710400}
I20250902 01:59:29.362713 1766 raft_consensus.cc:513] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.368589 1718 raft_consensus.cc:491] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.363577 1905 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.401135 30922 raft_consensus.cc:2408] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20250902 01:59:29.402829 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.370604 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:29.364538 1789 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250902 01:59:29.420472 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 51176.5us
W20250902 01:59:29.425668 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.413020 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2d2c9c23b6145c9b6601f7e78f1f43a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:29.389140 1741 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:29.365208 1804 leader_election.cc:290] T 5cf39ccba06f4a0998fb41dd68a0702a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:29.471256 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 104365us
W20250902 01:59:29.471295 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 51616.1us
W20250902 01:59:29.428892 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.473515 31038 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:29.474462 31727 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 5.
I20250902 01:59:29.474712 31067 raft_consensus.cc:2374] T a2d2c9c23b6145c9b6601f7e78f1f43a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5. Re-sending same reply.
I20250902 01:59:29.470419 30964 leader_election.cc:304] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:29.489833 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:29.511232 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.511894 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:29.512467 30993 raft_consensus.cc:2366] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 9. Current term is 10.
I20250902 01:59:29.513410 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.514272 30993 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:29.515714 30995 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.520711 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:29.522116 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f160859afd5945949c030026bfe3ac61" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.519896 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:29.522737 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.524497 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:29.356762 1961 consensus_queue.cc:1046] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20250902 01:59:29.385419 1794 raft_consensus.cc:2747] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:29.534454 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21e596cb1bea4b1c9015d9b8d9a188b6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:29.526062 31038 leader_election.cc:304] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:29.535262 30994 raft_consensus.cc:2391] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 088424002a894fd5bcc11a06af111445 in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:29.420189 1716 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250902 01:59:29.541929 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.544641 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de2e42094ddd41f99d12a4972ee07b9f" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:29.404584 1746 raft_consensus.cc:2802] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250902 01:59:29.420023 1954 raft_consensus.cc:491] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:29.406927 1929 raft_consensus.cc:2802] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:29.389379 1934 consensus_queue.cc:1035] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.145s
I20250902 01:59:29.549556 31067 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.391076 1874 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.473032 1789 raft_consensus.cc:491] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.478086 1701 raft_consensus.cc:2802] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 10 FOLLOWER]: Leader election won for term 10
I20250902 01:59:29.477294 1905 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.549170 31730 raft_consensus.cc:2466] T a2d2c9c23b6145c9b6601f7e78f1f43a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 5.
I20250902 01:59:29.525805 1686 raft_consensus.cc:2802] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:29.427234 1760 consensus_queue.cc:1035] T 63fa3c697d3e40afb9a01578cbe72700 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.131s
I20250902 01:59:29.558683 1949 raft_consensus.cc:2747] T 4ba5aa5dce214071a172ea365440f0dc P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250902 01:59:29.577898 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 87754.4us
I20250902 01:59:29.540956 1877 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:29.525903 1781 raft_consensus.cc:2802] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250902 01:59:29.554625 1954 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.580238 1789 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.554396 1965 raft_consensus.cc:491] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:29.487470 1718 raft_consensus.cc:513] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.525990 1834 consensus_queue.cc:1046] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.051s
I20250902 01:59:29.551713 1746 raft_consensus.cc:491] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:29.592602 1877 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.593341 1781 raft_consensus.cc:491] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.547454 1894 leader_election.cc:290] T e12bca11fa644c12b4c6f0658fa07084 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.558418 1960 raft_consensus.cc:3053] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:29.554633 1794 raft_consensus.cc:491] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:29.553151 1716 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.585754 1686 raft_consensus.cc:491] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.615114 1960 raft_consensus.cc:738] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:29.615386 1686 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.615307 1794 raft_consensus.cc:513] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.616027 1960 consensus_queue.cc:260] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.617162 1960 raft_consensus.cc:3058] T de2e42094ddd41f99d12a4972ee07b9f P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.617887 1794 leader_election.cc:290] T 22b1ce0ddf934a6baa56f3ee89e05309 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.541313 1860 consensus_queue.cc:260] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.615309 1716 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.622233 1686 raft_consensus.cc:513] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.593050 1701 raft_consensus.cc:695] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [term 10 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:29.611047 1746 raft_consensus.cc:3058] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:29.594200 1969 raft_consensus.cc:2802] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250902 01:59:29.629500 1686 leader_election.cc:290] T 50a1d33046ce49cd891b784fbe726d64 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.640321 1892 leader_election.cc:290] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.608626 1886 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.610731 1814 raft_consensus.cc:3053] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Stepping down as leader of term 4
I20250902 01:59:29.624517 1877 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.624805 1781 raft_consensus.cc:3058] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:29.744933 1766 leader_election.cc:290] T 02cd1389058944c1adc90f23d0fadb21 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:29.745337 1977 log.cc:927] Time spent T eecf760e0123488fb797b4457e0888d7 P 01f6731ce5044a6a948a5ae04da80e6f: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250902 01:59:29.745548 1929 raft_consensus.cc:695] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Leader. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
W20250902 01:59:29.746753 1975 log.cc:927] Time spent T 21e596cb1bea4b1c9015d9b8d9a188b6 P 088424002a894fd5bcc11a06af111445: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20250902 01:59:29.609133 1965 raft_consensus.cc:513] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.749531 1965 leader_election.cc:290] T cb3bd5884ec54e9da4138c559675467e P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:29.629769 1969 raft_consensus.cc:491] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:29.759860 1969 raft_consensus.cc:3058] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:29.746814 1929 consensus_queue.cc:237] T 2e3ca4f287b94f60b51ef1768df7bb70 P 088424002a894fd5bcc11a06af111445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
W20250902 01:59:29.763975 1967 log.cc:927] Time spent T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20250902 01:59:29.764037 1962 consensus_meta.cc:287] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.003s
W20250902 01:59:29.797513 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 418120us
W20250902 01:59:29.584228 31038 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 58229.2us
W20250902 01:59:29.575742 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.825024 1701 consensus_queue.cc:237] T 805e9f7636dc47059b5af8c090e5ac0a P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:29.802682 30917 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.031s sys 0.066s
W20250902 01:59:29.806228 31692 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 441766us
W20250902 01:59:29.831941 30917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.031s sys 0.067s
W20250902 01:59:29.807433 31693 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 216375us
W20250902 01:59:29.809260 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:29.558720 1417 consensus_meta.cc:287] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.510s user 0.007s sys 0.003s
I20250902 01:59:29.810402 1566 leader_election.cc:290] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.813453 31722 raft_consensus.cc:1238] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 7. Current term is 8. Ops: []
I20250902 01:59:29.710109 1814 raft_consensus.cc:738] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:29.801877 30892 leader_election.cc:304] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
W20250902 01:59:29.860598 1784 consensus_meta.cc:287] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f: Time spent flushing consensus metadata: real 0.789s user 0.009s sys 0.000s
I20250902 01:59:29.880578 31723 raft_consensus.cc:1273] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Refusing update from remote peer 975ecd397f5146409baac87e56a302e9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250902 01:59:29.881072 31723 pending_rounds.cc:85] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Aborting all ops after (but not including) 7
I20250902 01:59:29.881325 31723 pending_rounds.cc:107] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20250902 01:59:29.881551 31723 raft_consensus.cc:2887] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250902 01:59:29.883849 1814 consensus_queue.cc:260] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.885337 1814 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Advancing to term 5
W20250902 01:59:29.888444 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.892979 1934 raft_consensus.cc:491] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 088424002a894fd5bcc11a06af111445)
I20250902 01:59:29.894084 1934 raft_consensus.cc:513] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.895156 1960 raft_consensus.cc:491] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 975ecd397f5146409baac87e56a302e9)
I20250902 01:59:29.898795 1960 raft_consensus.cc:513] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.899390 30992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 2001) took 1670 ms. Trace:
W20250902 01:59:29.905802 30994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53758 (request call id 2010) took 1284 ms (client timeout 1530 ms). Trace:
W20250902 01:59:29.906426 30994 rpcz_store.cc:269] 0902 01:59:28.621294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.621395 (+ 101us) service_pool.cc:225] Handling call
0902 01:59:29.905325 (+1283930us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b5400127854. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.905601 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1137662848}
I20250902 01:59:29.908437 31069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46548 (request call id 1771) took 4068 ms. Trace:
I20250902 01:59:29.908883 31069 rpcz_store.cc:276] 0902 01:59:25.839472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.045937 (+3206465us) service_pool.cc:225] Handling call
0902 01:59:29.046177 (+ 240us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.354767 (+308590us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.907850 (+553083us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400020654. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:29.908217 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":404931456}
I20250902 01:59:29.909940 30999 raft_consensus.cc:1238] T 45b3012a0ae843acbfd03ed33966d585 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 4372b4c4c866431faa1bf3e91fd47a88 for earlier term 6. Current term is 7. Ops: []
I20250902 01:59:29.911957 1951 raft_consensus.cc:491] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.912973 1951 raft_consensus.cc:513] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.915906 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:29.916906 31066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1769) took 4077 ms (client timeout 3371 ms). Trace:
W20250902 01:59:29.917438 31066 rpcz_store.cc:269] 0902 01:59:25.838964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:27.165028 (+1326064us) service_pool.cc:225] Handling call
0902 01:59:29.916454 (+2751426us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b5400157a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.916636 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5045723520}
I20250902 01:59:29.878074 1784 raft_consensus.cc:513] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.920315 1784 leader_election.cc:290] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:29.921343 31068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38110 (request call id 22) took 3116 ms. Trace:
I20250902 01:59:29.921697 31068 rpcz_store.cc:276] 0902 01:59:26.816001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:27.425238 (+609237us) service_pool.cc:225] Handling call
0902 01:59:29.920952 (+2495714us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400157a54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212a813 00007fc6070901ba 00007fc60072a1c4 00007fc60072a152 00007fc60072a081 00007fc600729ffc 00007fc6007292b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.921129 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4407900928}
I20250902 01:59:29.923241 31724 raft_consensus.cc:2391] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4372b4c4c866431faa1bf3e91fd47a88 in current term 7: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:29.923727 30921 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 4372b4c4c866431faa1bf3e91fd47a88 [term 10 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 9. Current term is 10. Ops: []
I20250902 01:59:29.924475 31724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51240 (request call id 716) took 1302 ms. Trace:
I20250902 01:59:29.924866 31724 rpcz_store.cc:276] 0902 01:59:28.622365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.622494 (+ 129us) service_pool.cc:225] Handling call
0902 01:59:29.924201 (+1301707us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b54005901d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.924371 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1998103424}
I20250902 01:59:29.925462 30925 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: []
W20250902 01:59:29.918080 31072 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1780) took 4040 ms (client timeout 1910 ms). Trace:
W20250902 01:59:29.926047 31072 rpcz_store.cc:269] 0902 01:59:25.877396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.082144 (+204748us) service_pool.cc:225] Handling call
0902 01:59:29.917700 (+3835556us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400258854. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.917886 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3209949056}
W20250902 01:59:29.928505 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:29.929026 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:29.929694 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.931051 30889 leader_election.cc:304] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.931090 1982 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:29.933024 1982 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.933743 30922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.934590 30922 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:29.936878 1960 leader_election.cc:290] T ade302bdd69a4fa7876c90a1fbd87b68 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:29.931201 30998 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53758 (request call id 2004) took 1589 ms. Trace:
I20250902 01:59:29.899971 30992 rpcz_store.cc:276] 0902 01:59:28.276650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.276802 (+ 152us) service_pool.cc:225] Handling call
0902 01:59:29.898794 (+1621992us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540004f6d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:29.899070 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":518410624}
I20250902 01:59:29.946220 30925 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
I20250902 01:59:29.946544 1566 raft_consensus.cc:3058] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 4 FOLLOWER]: Advancing to term 5
I20250902 01:59:29.947114 30925 raft_consensus.cc:2391] T 02cd1389058944c1adc90f23d0fadb21 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 8: Already voted for candidate 4372b4c4c866431faa1bf3e91fd47a88 in this term.
W20250902 01:59:29.949005 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.950026 1411 leader_election.cc:290] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:29.954404 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:29.920445 1951 leader_election.cc:290] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:29.939677 30998 rpcz_store.cc:276] 0902 01:59:28.341363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.341521 (+ 158us) service_pool.cc:225] Handling call
0902 01:59:28.367996 (+ 26475us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:28.528588 (+160592us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:28.528711 (+ 123us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:28.528854 (+ 143us) raft_consensus.cc:1643] Marking committed up to 3
0902 01:59:28.528980 (+ 126us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.930654 (+1401674us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400178754. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:29.930725 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:29.930997 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":343604480}
I20250902 01:59:29.966003 30997 raft_consensus.cc:1238] T 0c93c2499f7743579a074e5f2646ad2d P 975ecd397f5146409baac87e56a302e9 [term 8 LEADER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20250902 01:59:29.968329 1955 raft_consensus.cc:2802] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader election won for term 5
I20250902 01:59:29.969144 1955 raft_consensus.cc:695] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:29.972054 31071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 693) took 3924 ms. Trace:
I20250902 01:59:29.972405 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
I20250902 01:59:29.973140 30921 raft_consensus.cc:3058] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:29.973345 1718 leader_election.cc:290] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
W20250902 01:59:29.962837 30892 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 161355us
I20250902 01:59:29.984741 1716 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.986305 30996 raft_consensus.cc:2391] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:29.988411 30892 leader_election.cc:304] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:29.989472 1830 consensus_queue.cc:260] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:29.990635 1665 raft_consensus.cc:2747] T 21e596cb1bea4b1c9015d9b8d9a188b6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250902 01:59:29.990813 1830 raft_consensus.cc:3058] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:29.991008 30921 raft_consensus.cc:2466] T 26ad5d89b038487b8bbc776913d5a215 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:29.992605 1982 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:29.994002 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 63172us
I20250902 01:59:29.954538 1969 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:29.970755 1955 consensus_queue.cc:237] T a2d2c9c23b6145c9b6601f7e78f1f43a P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:30.006255 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.006975 1934 leader_election.cc:290] T 543054d6241649d59bb40156217a73bc P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:30.007747 1718 raft_consensus.cc:491] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:30.006644 1789 raft_consensus.cc:513] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:30.013499 30921 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88" is_pre_election: true
W20250902 01:59:30.025705 1893 consensus_meta.cc:287] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.914s user 0.013s sys 0.002s
I20250902 01:59:30.026142 1893 raft_consensus.cc:513] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:30.045254 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.027765 30889 leader_election.cc:304] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
I20250902 01:59:30.021217 1718 raft_consensus.cc:513] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:30.214222 30920 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.006s sys 0.009s
I20250902 01:59:30.051124 30924 raft_consensus.cc:1238] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 975ecd397f5146409baac87e56a302e9 for earlier term 4. Current term is 6. Ops: []
I20250902 01:59:29.972411 31071 rpcz_store.cc:276] 0902 01:59:26.047628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.276741 (+3229113us) service_pool.cc:225] Handling call
0902 01:59:29.277111 (+ 370us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.277337 (+ 226us) raft_consensus.cc:1532] Early marking committed up to index 8
0902 01:59:29.312191 (+ 34854us) log.cc:844] Serialized 14 byte log entry
0902 01:59:29.903343 (+591152us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:29.961173 (+ 57830us) raft_consensus.cc:1643] Marking committed up to 8
0902 01:59:29.971653 (+ 10480us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.971751 (+ 98us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:29.971885 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40067,"prepare.queue_time_us":35265,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":2747,"raft.queue_time_us":45692,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":4458,"spinlock_wait_cycles":7202432,"thread_start_us":550730,"threads_started":2,"wal-append.queue_time_us":293220}
W20250902 01:59:30.092087 30966 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:30.054991 31039 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:44457, user_credentials={real_user=slave}} blocked reactor thread for 51545.2us
W20250902 01:59:30.254334 30920 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.863s user 0.007s sys 0.010s
W20250902 01:59:30.259877 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.219146 1877 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.270975 1877 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:30.272758 1799 leader_election.cc:290] T 2925593648194df5b7741d448a047401 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:30.049495 1789 leader_election.cc:290] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:30.077651 1716 leader_election.cc:290] T f160859afd5945949c030026bfe3ac61 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
W20250902 01:59:30.291640 1792 consensus_meta.cc:287] T 9685199b6c7b42f391ab48fdf86d71ad P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.833s user 0.006s sys 0.008s
I20250902 01:59:30.292150 1665 raft_consensus.cc:2747] T 4120ecbc46e34f93bb86004798318c08 P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250902 01:59:30.293021 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45b3012a0ae843acbfd03ed33966d585" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.293754 31071 raft_consensus.cc:3058] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
W20250902 01:59:30.294246 30967 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.296223 1853 consensus_queue.cc:1046] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20250902 01:59:30.296538 30999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:30.297533 30999 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.296665 30994 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:30.297559 1853 raft_consensus.cc:3053] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Stepping down as leader of term 9
I20250902 01:59:30.297883 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dad02ef06fef44c5b5ecc42134c43091" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.298233 1853 raft_consensus.cc:738] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:30.298758 30992 raft_consensus.cc:2466] T dad02ef06fef44c5b5ecc42134c43091 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.300992 31695 leader_election.cc:304] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 088424002a894fd5bcc11a06af111445; no voters:
I20250902 01:59:30.302415 1701 raft_consensus.cc:2802] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:30.302783 1701 raft_consensus.cc:491] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250902 01:59:30.303328 30992 raft_consensus.cc:1238] T 38649bfc334c4db6921eb3b12130e10d P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20250902 01:59:30.305431 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.303144 1701 raft_consensus.cc:3058] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:30.317034 30998 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:30.328048 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.328967 30992 raft_consensus.cc:2391] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 10: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:30.329949 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:30.334782 1781 consensus_meta.cc:287] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.584s user 0.012s sys 0.004s
I20250902 01:59:30.327296 1837 consensus_queue.cc:1046] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.009s
I20250902 01:59:30.336532 1837 raft_consensus.cc:3053] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:30.336889 1837 raft_consensus.cc:738] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:30.337252 30925 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9c9ad89315794238b432653557334854"
dest_uuid: "4372b4c4c866431faa1bf3e91fd47a88"
mode: GRACEFUL
new_leader_uuid: "975ecd397f5146409baac87e56a302e9"
from {username='slave'} at 127.0.0.1:34128
I20250902 01:59:30.337760 1837 consensus_queue.cc:260] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.339214 1969 leader_election.cc:290] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 6 election: Requested vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:30.338192 30925 raft_consensus.cc:604] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Received request to transfer leadership to TS 975ecd397f5146409baac87e56a302e9
I20250902 01:59:30.340044 30925 raft_consensus.cc:612] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250902 01:59:30.335352 1781 raft_consensus.cc:513] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.350605 31693 leader_election.cc:304] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f6731ce5044a6a948a5ae04da80e6f, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:30.299288 1853 consensus_queue.cc:260] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
W20250902 01:59:30.353927 31067 consensus_meta.cc:287] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445: Time spent flushing consensus metadata: real 0.692s user 0.011s sys 0.003s
W20250902 01:59:30.366088 31070 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46548 (request call id 1778) took 4510 ms (client timeout 2058 ms). Trace:
I20250902 01:59:30.366410 31001 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 9: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
W20250902 01:59:30.365432 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 352624us
I20250902 01:59:30.371394 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:30.366510 31070 rpcz_store.cc:269] 0902 01:59:25.855765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:26.198839 (+343074us) service_pool.cc:225] Handling call
0902 01:59:30.355177 (+4156338us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540036ef54. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:30.365928 (+ 10751us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3627548544}
I20250902 01:59:30.372845 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.373490 31070 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8.
I20250902 01:59:30.374423 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4120ecbc46e34f93bb86004798318c08" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:30.378463 31041 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.375927 1766 raft_consensus.cc:2802] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:30.385731 1856 leader_election.cc:290] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
I20250902 01:59:30.386801 30998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:30.391690 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.394606 31067 raft_consensus.cc:2466] T de2e42094ddd41f99d12a4972ee07b9f P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 7.
I20250902 01:59:30.378650 1837 raft_consensus.cc:3058] T 38649bfc334c4db6921eb3b12130e10d P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:30.403458 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "63fa3c697d3e40afb9a01578cbe72700" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.410285 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.394824 1766 raft_consensus.cc:695] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Becoming Leader. State: Replica: 01f6731ce5044a6a948a5ae04da80e6f, State: Running, Role: LEADER
I20250902 01:59:30.411054 31070 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:30.411782 1853 raft_consensus.cc:3058] T b03cb35549a346d28baa21d708d665e2 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Advancing to term 10
I20250902 01:59:30.411959 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.412163 1766 consensus_queue.cc:237] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.412551 31070 raft_consensus.cc:2466] T aeb7c457a3f741b5b4d057f06363c614 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
W20250902 01:59:30.415932 1954 consensus_meta.cc:287] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.743s user 0.005s sys 0.007s
I20250902 01:59:30.416275 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.416395 1954 raft_consensus.cc:513] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.417050 31070 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:30.418110 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:30.418808 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.419179 1954 leader_election.cc:290] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:30.420485 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aeb7c457a3f741b5b4d057f06363c614" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:30.420818 31040 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.418807 31070 raft_consensus.cc:2391] T aa4b903a93464946ae238ce634b483d0 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 8: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
I20250902 01:59:30.421236 30993 raft_consensus.cc:2391] T aeb7c457a3f741b5b4d057f06363c614 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:30.422333 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ade302bdd69a4fa7876c90a1fbd87b68" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.422740 31074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45756 (request call id 690) took 4747 ms. Trace:
I20250902 01:59:30.423390 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "805e9f7636dc47059b5af8c090e5ac0a" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:30.424546 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.425239 30993 raft_consensus.cc:2466] T d920df318de242f797c216f655b63218 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.425943 31067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46556 (request call id 1992) took 1030 ms. Trace:
I20250902 01:59:30.426549 31067 rpcz_store.cc:276] 0902 01:59:29.377423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.544386 (+166963us) service_pool.cc:225] Handling call
0902 01:59:30.402907 (+858521us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250902 01:59:30.421057 1781 leader_election.cc:290] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [CANDIDATE]: Term 9 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:30.423084 31074 rpcz_store.cc:276] 0902 01:59:25.675444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:28.967979 (+3292535us) service_pool.cc:225] Handling call
0902 01:59:29.003523 (+ 35544us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.364388 (+360865us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:30.422344 (+1057956us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540024b3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:30.422565 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":712844544}
W20250902 01:59:30.432844 31693 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.436605 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.437614 30992 raft_consensus.cc:2466] T 568a81baa7ef422eb8125c6e8bb9813b P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 7.
I20250902 01:59:30.437960 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.438158 1765 leader_election.cc:290] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Requested vote from peers 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531), 01f6731ce5044a6a948a5ae04da80e6f (127.29.21.196:44285)
I20250902 01:59:30.435078 1718 leader_election.cc:290] T 0c93c2499f7743579a074e5f2646ad2d P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:30.438788 30992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.438791 31067 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 10.
I20250902 01:59:30.440378 31067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02cd1389058944c1adc90f23d0fadb21" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.441120 30997 raft_consensus.cc:1238] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 8. Current term is 9. Ops: []
I20250902 01:59:30.441236 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.442210 31001 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 5.
I20250902 01:59:30.443042 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
I20250902 01:59:30.443589 30997 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:30.444799 30892 leader_election.cc:304] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:30.443234 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.439030 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.453464 31070 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 088424002a894fd5bcc11a06af111445 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20250902 01:59:30.454124 30999 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.454955 30889 leader_election.cc:304] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88; no voters: 088424002a894fd5bcc11a06af111445, 975ecd397f5146409baac87e56a302e9
W20250902 01:59:30.456046 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.462757 1954 raft_consensus.cc:2747] T 558db87de6fa4361bf0ca91e27d34943 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250902 01:59:30.466467 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.469390 30997 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 975ecd397f5146409baac87e56a302e9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:30.470522 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.471735 30925 raft_consensus.cc:1238] T 122e9f7b6c1d4295b7d12445ed0116ed P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 5. Current term is 6. Ops: []
W20250902 01:59:30.472898 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.469623 1704 raft_consensus.cc:491] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250902 01:59:30.482722 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.482510 1704 raft_consensus.cc:513] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.483366 31724 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.483919 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ede06f18c6574c57857475151e3767d2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
W20250902 01:59:30.484946 31695 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:40531, user_credentials={real_user=slave}} blocked reactor thread for 115785us
I20250902 01:59:30.486322 30997 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb3bd5884ec54e9da4138c559675467e" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.456593 1856 raft_consensus.cc:491] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4372b4c4c866431faa1bf3e91fd47a88)
I20250902 01:59:30.488200 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c9ad89315794238b432653557334854" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.487028 1765 raft_consensus.cc:2802] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Leader election won for term 7
I20250902 01:59:30.490386 31000 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "703afdc575144ab78740c24ef2852ca8" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
W20250902 01:59:30.500223 31039 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250902 01:59:30.501112 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 92664.9us
W20250902 01:59:30.502982 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.505194 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.505520 31074 raft_consensus.cc:3058] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:30.491317 1704 leader_election.cc:290] T d8ce2ebbb1c24a6c998649ed3656707a P 01f6731ce5044a6a948a5ae04da80e6f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 088424002a894fd5bcc11a06af111445 (127.29.21.195:40531)
I20250902 01:59:30.510758 31070 raft_consensus.cc:2466] T ede06f18c6574c57857475151e3767d2 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 8.
I20250902 01:59:30.511780 30993 raft_consensus.cc:2466] T 92f16b8a15144fcf8807608bdee71123 P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 4.
I20250902 01:59:30.514432 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.514684 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.515106 30993 raft_consensus.cc:2391] T 8f5d81000607410da8880ef8e5517680 P 975ecd397f5146409baac87e56a302e9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 9: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:30.515174 1909 raft_consensus.cc:3058] T d920df318de242f797c216f655b63218 P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.515321 31727 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.510583 30997 raft_consensus.cc:2466] T cb3bd5884ec54e9da4138c559675467e P 975ecd397f5146409baac87e56a302e9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 4.
W20250902 01:59:30.518221 31694 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.516703 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e3ca4f287b94f60b51ef1768df7bb70" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.515805 30993 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa4b903a93464946ae238ce634b483d0" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.522027 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.524708 31724 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:30.525305 31724 raft_consensus.cc:3058] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 6 LEARNER]: Advancing to term 8
I20250902 01:59:30.525998 30996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54174 (request call id 555) took 1013 ms. Trace:
I20250902 01:59:30.526293 30996 rpcz_store.cc:276] 0902 01:59:29.512136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.521868 (+ 9732us) service_pool.cc:225] Handling call
0902 01:59:30.525822 (+1003954us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54002dc3d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc6021342aa 00007fc60708d5c7 00007fc60071c0c4 00007fc60071c052 00007fc60071bf81 00007fc60071befc 00007fc60071b1b2 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:30.525945 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1065084672}
I20250902 01:59:30.526361 31725 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:30.527343 31725 raft_consensus.cc:3058] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Advancing to term 9
I20250902 01:59:30.505738 1765 raft_consensus.cc:695] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [term 7 LEADER]: Becoming Leader. State: Replica: 4372b4c4c866431faa1bf3e91fd47a88, State: Running, Role: LEADER
I20250902 01:59:30.531446 31000 raft_consensus.cc:2466] T 703afdc575144ab78740c24ef2852ca8 P 975ecd397f5146409baac87e56a302e9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.531879 31727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0692c5029b5e4ab088324d038ff964be" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.532680 31727 raft_consensus.cc:2466] T 0692c5029b5e4ab088324d038ff964be P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:30.521631 1856 raft_consensus.cc:513] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:30.534543 1856 leader_election.cc:290] T cba8e025cdf94c449fa56870398f3419 P 088424002a894fd5bcc11a06af111445 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4372b4c4c866431faa1bf3e91fd47a88 (127.29.21.193:44457), 975ecd397f5146409baac87e56a302e9 (127.29.21.194:46439)
W20250902 01:59:30.534624 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.535827 1883 raft_consensus.cc:3053] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Stepping down as leader of term 6
I20250902 01:59:30.536077 31071 raft_consensus.cc:2466] T 45b3012a0ae843acbfd03ed33966d585 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:30.536381 1883 raft_consensus.cc:738] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 088424002a894fd5bcc11a06af111445, State: Running, Role: LEADER
I20250902 01:59:30.537073 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a61d1c51a3034d52aa9a91fea8525fc7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.537767 1883 consensus_queue.cc:260] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } }
I20250902 01:59:30.538862 1883 raft_consensus.cc:3058] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:30.540354 31722 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51234 (request call id 1038) took 1457 ms. Trace:
I20250902 01:59:30.540791 31722 rpcz_store.cc:276] 0902 01:59:29.082936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.397980 (+315044us) service_pool.cc:225] Handling call
0902 01:59:29.468662 (+ 70682us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.926287 (+457625us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:30.540065 (+613778us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54006ae0d4. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc602130dc1 00007fc60212f7fe 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106
0902 01:59:30.540277 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":791326464}
I20250902 01:59:30.543568 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f5d81000607410da8880ef8e5517680" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.545130 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "151f832cfd4e42d3a547a7cef6bb995f" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.545856 31068 raft_consensus.cc:2466] T 151f832cfd4e42d3a547a7cef6bb995f P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6731ce5044a6a948a5ae04da80e6f in term 6.
I20250902 01:59:30.537755 31071 raft_consensus.cc:2366] T a61d1c51a3034d52aa9a91fea8525fc7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for earlier term 5. Current term is 6.
I20250902 01:59:30.546896 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d920df318de242f797c216f655b63218" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.547000 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eecf760e0123488fb797b4457e0888d7" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.547843 31071 raft_consensus.cc:2466] T eecf760e0123488fb797b4457e0888d7 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:30.548442 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48e0a90ceb7a486aa3a93a73f087dfc6" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.548713 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.549278 31069 raft_consensus.cc:2466] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 9.
I20250902 01:59:30.549377 31071 raft_consensus.cc:2466] T 5cf39ccba06f4a0998fb41dd68a0702a P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:30.550500 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92f16b8a15144fcf8807608bdee71123" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.550763 30892 leader_election.cc:304] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:30.551308 31071 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a3db1aaea8d4a3eba2de840361e0849" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.551573 1955 raft_consensus.cc:2802] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250902 01:59:30.551895 31071 raft_consensus.cc:3058] T 5a3db1aaea8d4a3eba2de840361e0849 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.552042 1955 raft_consensus.cc:491] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:30.552513 1955 raft_consensus.cc:3058] T 48e0a90ceb7a486aa3a93a73f087dfc6 P 4372b4c4c866431faa1bf3e91fd47a88 [term 9 FOLLOWER]: Advancing to term 10
W20250902 01:59:30.528407 1566 consensus_meta.cc:287] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.577s user 0.011s sys 0.000s
I20250902 01:59:30.555704 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c9675a8370e4d4a9f95b3c568abe972" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.505086 1781 consensus_queue.cc:1046] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20250902 01:59:30.559804 31074 raft_consensus.cc:2466] T 9c9ad89315794238b432653557334854 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4372b4c4c866431faa1bf3e91fd47a88 in term 6.
I20250902 01:59:30.559883 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543054d6241649d59bb40156217a73bc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.561331 31722 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e12bca11fa644c12b4c6f0658fa07084" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 8 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:30.563383 30892 leader_election.cc:304] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088424002a894fd5bcc11a06af111445, 4372b4c4c866431faa1bf3e91fd47a88; no voters:
I20250902 01:59:30.565810 31727 raft_consensus.cc:1238] T 26ad5d89b038487b8bbc776913d5a215 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 LEADER]: Rejecting Update request from peer 088424002a894fd5bcc11a06af111445 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20250902 01:59:30.563293 30996 raft_consensus.cc:1238] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Rejecting Update request from peer 01f6731ce5044a6a948a5ae04da80e6f for earlier term 9. Current term is 10. Ops: []
I20250902 01:59:30.565229 31001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.566862 31070 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "122e9f7b6c1d4295b7d12445ed0116ed" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.567773 31722 raft_consensus.cc:3058] T e12bca11fa644c12b4c6f0658fa07084 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.568800 31069 raft_consensus.cc:2391] T 6c9675a8370e4d4a9f95b3c568abe972 P 088424002a894fd5bcc11a06af111445 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 9: Already voted for candidate 088424002a894fd5bcc11a06af111445 in this term.
W20250902 01:59:30.571343 31692 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.560020 1566 raft_consensus.cc:2747] T 9685199b6c7b42f391ab48fdf86d71ad P 4372b4c4c866431faa1bf3e91fd47a88 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250902 01:59:30.572733 31001 raft_consensus.cc:2466] T 416020230983423c9f843070a7893658 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 10.
I20250902 01:59:30.574834 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22b1ce0ddf934a6baa56f3ee89e05309" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.578217 31074 raft_consensus.cc:2466] T 22b1ce0ddf934a6baa56f3ee89e05309 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 7.
I20250902 01:59:30.579463 31070 raft_consensus.cc:3058] T 122e9f7b6c1d4295b7d12445ed0116ed P 088424002a894fd5bcc11a06af111445 [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:30.577787 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b03cb35549a346d28baa21d708d665e2" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9"
W20250902 01:59:30.575977 1905 consensus_meta.cc:287] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.911s user 0.007s sys 0.008s
I20250902 01:59:30.584128 31000 raft_consensus.cc:3053] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 LEADER]: Stepping down as leader of term 9
I20250902 01:59:30.584379 31000 raft_consensus.cc:738] T 805e9f7636dc47059b5af8c090e5ac0a P 975ecd397f5146409baac87e56a302e9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 975ecd397f5146409baac87e56a302e9, State: Running, Role: LEADER
I20250902 01:59:30.580312 31068 raft_consensus.cc:2374] T d920df318de242f797c216f655b63218 P 088424002a894fd5bcc11a06af111445 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 8. Re-sending same reply.
I20250902 01:59:30.587255 30996 raft_consensus.cc:2366] T b03cb35549a346d28baa21d708d665e2 P 975ecd397f5146409baac87e56a302e9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f for earlier term 9. Current term is 10.
W20250902 01:59:30.588233 31695 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250902 01:59:30.536001 1765 consensus_queue.cc:237] T de2e42094ddd41f99d12a4972ee07b9f P 4372b4c4c866431faa1bf3e91fd47a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.589112 31728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.578842 1954 raft_consensus.cc:2802] T 9c9ad89315794238b432653557334854 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader election won for term 6
I20250902 01:59:30.577442 1781 consensus_queue.cc:1230] T e12bca11fa644c12b4c6f0658fa07084 P 975ecd397f5146409baac87e56a302e9 [LEADER]: Peer 088424002a894fd5bcc11a06af111445 log is divergent from this leader: its last log entry 6.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20250902 01:59:30.595685 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c93c2499f7743579a074e5f2646ad2d" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.596771 31069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59bc91ec50764c7bb821a4288bea8d16" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.599206 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50a1d33046ce49cd891b784fbe726d64" candidate_uuid: "088424002a894fd5bcc11a06af111445" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f"
I20250902 01:59:30.599964 31721 raft_consensus.cc:3058] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 5 FOLLOWER]: Advancing to term 6
I20250902 01:59:30.609081 31074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ba5aa5dce214071a172ea365440f0dc" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.611150 31068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26ad5d89b038487b8bbc776913d5a215" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445"
I20250902 01:59:30.608601 31725 raft_consensus.cc:2466] T 6c9675a8370e4d4a9f95b3c568abe972 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 9.
W20250902 01:59:30.598660 1741 consensus_meta.cc:287] T 6c9675a8370e4d4a9f95b3c568abe972 P 975ecd397f5146409baac87e56a302e9: Time spent flushing consensus metadata: real 0.987s user 0.004s sys 0.014s
I20250902 01:59:30.613997 31721 raft_consensus.cc:2466] T 50a1d33046ce49cd891b784fbe726d64 P 01f6731ce5044a6a948a5ae04da80e6f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 6.
I20250902 01:59:30.611673 31069 raft_consensus.cc:2466] T 59bc91ec50764c7bb821a4288bea8d16 P 088424002a894fd5bcc11a06af111445 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975ecd397f5146409baac87e56a302e9 in term 6.
I20250902 01:59:30.614583 30918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38810 (request call id 2010) took 2825 ms. Trace:
W20250902 01:59:30.617458 30889 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:46439, user_credentials={real_user=slave}} blocked reactor thread for 63332.6us
W20250902 01:59:30.607648 1746 consensus_meta.cc:287] T 24426e817ea94ac5a7014c3457ea087c P 4372b4c4c866431faa1bf3e91fd47a88: Time spent flushing consensus metadata: real 0.888s user 0.009s sys 0.009s
I20250902 01:59:30.620028 30996 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2925593648194df5b7741d448a047401" candidate_uuid: "01f6731ce5044a6a948a5ae04da80e6f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "975ecd397f5146409baac87e56a302e9" is_pre_election: true
I20250902 01:59:30.586862 1701 raft_consensus.cc:513] T f160859afd5945949c030026bfe3ac61 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 40531 } } peers { permanent_uuid: "01f6731ce5044a6a948a5ae04da80e6f" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 44285 } attrs { promote: false } }
I20250902 01:59:30.631599 30889 leader_election.cc:304] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4372b4c4c866431faa1bf3e91fd47a88, 975ecd397f5146409baac87e56a302e9; no voters:
I20250902 01:59:30.631887 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0f24bac96dc42dba9adc1f6679c0d48" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.632500 1789 raft_consensus.cc:2802] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250902 01:59:30.632920 1789 raft_consensus.cc:491] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 01f6731ce5044a6a948a5ae04da80e6f)
I20250902 01:59:30.633060 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "568a81baa7ef422eb8125c6e8bb9813b" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.633325 1789 raft_consensus.cc:3058] T d7f66bcc25964883baf67f48889ec766 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Advancing to term 7
I20250902 01:59:30.633832 31721 raft_consensus.cc:2391] T 568a81baa7ef422eb8125c6e8bb9813b P 01f6731ce5044a6a948a5ae04da80e6f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 in current term 8: Already voted for candidate 01f6731ce5044a6a948a5ae04da80e6f in this term.
I20250902 01:59:30.634557 31721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "416020230983423c9f843070a7893658" candidate_uuid: "975ecd397f5146409baac87e56a302e9" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "01f6731ce5044a6a948a5ae04da80e6f" is_pre_election: true
I20250902 01:59:30.634737 30995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54174 (request call id 563) took 1120 ms. Trace:
I20250902 01:59:30.635257 31721 raft_consensus.cc:2408] T 416020230983423c9f843070a7893658 P 01f6731ce5044a6a948a5ae04da80e6f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 975ecd397f5146409baac87e56a302e9 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20250902 01:59:30.624004 30996 raft_consensus.cc:2391] T 2925593648194df5b7741d448a047401 P 975ecd397f5146409baac87e56a302e9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01f6731ce5044a6a948a5ae04da80e6f in current term 7: Already voted for candidate 975ecd397f5146409baac87e56a302e9 in this term.
I20250902 01:59:30.626860 1860 raft_consensus.cc:3058] T 02cd1389058944c1adc90f23d0fadb21 P 088424002a894fd5bcc11a06af111445 [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.628401 31072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cf39ccba06f4a0998fb41dd68a0702a" candidate_uuid: "4372b4c4c866431faa1bf3e91fd47a88" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "088424002a894fd5bcc11a06af111445" is_pre_election: true
I20250902 01:59:30.629705 1763 raft_consensus.cc:3058] T 818cd33c83fe427a981bf0c807af5b14 P 01f6731ce5044a6a948a5ae04da80e6f [term 7 FOLLOWER]: Advancing to term 8
I20250902 01:59:30.647902 31724 raft_consensus.cc:2466] T 4ba5aa5dce214071a172ea365440f0dc P 01f6731ce5044a6a948a5ae04da80e6f [term 8 LEARNER]: Leader election vote request: Granting yes vote for candidate 088424002a894fd5bcc11a06af111445 in term 8.
I20250902 01:59:30.635042 30995 rpcz_store.cc:276] 0902 01:59:29.514342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0902 01:59:29.547583 (+ 33241us) service_pool.cc:225] Handling call
0902 01:59:29.904136 (+356553us) raft_consensus.cc:1469] Updating replica for 0 ops
0902 01:59:29.904706 (+ 570us) raft_consensus.cc:1532] Early marking committed up to index 0
0902 01:59:29.904740 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0902 01:59:29.904818 (+ 78us) raft_consensus.cc:1643] Marking committed up to 6
0902 01:59:29.904860 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0902 01:59:29.904930 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0902 01:59:30.634500 (+729570us) spinlock_profiling.cc:243] Waited 88 ms on lock 0x7b540004f6d0. stack: 00007fc5ff19c823 00007fc5ff19c720 00007fc5feaf6574 00007fc6085f8784 00007fc6085f8719 00007fc6020b46e1 00007fc60212f86d 00007fc60708cc89 00007fc600719b44 00007fc600719ad2 00007fc600719a01 00007fc60071997c 00007fc600718c32 00007fc60052fccc 00007fc60052f106 00007fc60052eaaf
0902 01:59:30.634592 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202454400}
I20250902 01:59:30.592056 1905 raft_consensus.cc:513] T a61d1c51a3034d52aa9a91fea8525fc7 P 4372b4c4c866431faa1bf3e91fd47a88 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4372b4c4c866431faa1bf3e91fd47a88" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 44457 } } peers { permanent_uuid: "975ecd397f5146409baac87e56a302e9" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 46439 } } peers { permanent_uuid: "088424002a894fd5bcc11a06af111445" member_type: VOTER last_known_addr { host: "127.29.21.195"